πŸ™Š sharing 8 super useful Python Programming suggestions

Time:2020-1-7

When we use Python for machine learning and modeling projects, everyone will have their own set of project file management habits, and I also have a set of methods, which are summed up by the pits we have stepped on. Now I’ll share with you, hoping that some places can be used for reference.

See it first

  • File project documents in advance
  • Never manually modify the source data and make a backup
  • Properly configure the path
  • Make notes and explanations where necessary
  • Speed up your Python loop code
  • Visualize your loop code progress
  • Use efficient exception capture tools
  • Think more about code robustness
  1. File project documents in advance

Every time I start a new job, I used to be greedy for convenience. Code, data and documents were all put in one folder, which seemed very messy. It once made the backtracking process very painful. Or when I changed a computer, all the files could not run. I needed to modify the path by myself, which was very painful.

After your own exploration, you can roughly divide the project into several subfolders, code in the main folder:

πŸ™Š sharing 8 super useful Python Programming suggestions

  1. Never manually modify the source data and make a backup

We need to back up the source data to facilitate our next backtracking. We can carry out the next operation or modify the intermediate steps. In addition, we need to back up the code and other files to avoid accidental loss.

Here comes fromGood LinuxFour tools are recommended:

  • Git version control system
  • Rsync file backup
  • Dropbox cloud storage
  • Time machine

For more tool introduction and use, I will not expand it. You can go to learn about it by yourself.

  1. Properly configure the path

Many students like to use the absolute path directly when writing the path. Although in general, there is no problem, if the code is shared to other people for learning or running, the problem will come. In many cases, it can’t run directly,

It is suggested that:

  • Use relative path: the script is located in the main directory, and other resources (such as data, third-party packages, etc.) are located in the same level or low-level directory, such as. / data / processed / test1.csv
  • Global path configuration variable:
#Set home directory
HOME_PATH = r'E:\ML0615- PROJECT1'

#Read data
data = open(HOME_PATH+'/data/processed/test1.csv') 
data = pd.read_csv(data)
data.head()
  1. Make notes and explanations where necessary

I believe most people feel the same thing, don’t they? Take back the code you wrote a month ago and see how much you can read (if you don’t make notes)

  1. Speed up your Python loop code

Here’s an article from Brother Yun: 24 ways to speed up your Python:

https://mp.weixin.qq.com/s/8b…

Collect it, read it many times, and form a good habit, so that you can write code faster and faster

  1. Visualize your loop code progress

Here is a python library, tqdm. Install it first:pip install tqdm

This is a library that can show the progress of the cycle. With it, you can plan more.

You can see the following example:

πŸ™Š sharing 8 super useful Python Programming suggestions

  1. Use efficient exception capture tools

In the past, I used to go all the way with a print() function. Although there is no problem, the efficiency is still relatively slow. Later, I found a decorator called pysnooper, as if I found a new continent.

We usually debug in places where we may think there is a problem. Go to print and output, see what the actual output is, and then think about the problem. This requires us to change the code very carefully. Compared with adding a decorator directly, it is very troublesome.

You can take a look at example:

import pysnooper

@pysnooper.snoop('./log/file.log')
def number_to_bits(number):
    if number:
        bits = []
        while number:
            number, remainder = divmod(number, 2)
            bits.insert(0, remainder)
        return bits
    else:
        return [0]

number_to_bits(6)

We save the output of each step of the function as file.log, and we can directly see what’s wrong.

πŸ™Š sharing 8 super useful Python Programming suggestions

πŸ“š project address: https://github.com/cool-rr/py
https://mp.weixin.qq.com/s/zq…

  1. Think more about code robustness

As the name implies, what is the robustness of code is that it can withstand the test of various exception scenarios. Exception handling consists of two parts: “catch” and “throw”. “Capture” refers to the use of try… Except to wrap specific statements and properly complete the error process processing. The proper use of raise to actively “throw” exceptions is an essential part of elegant code. Here are some summary points for your reference:

1) Know what parameters to pass in, type, number… (exception handling, logical judgment)

def add(a, b):
  if isinstance(a, int) and isinstance(b, int):
      return a+b
  else:
      Return 'wrong parameter type'

print(add(1, 2))
print(add(1, 'a'))

2) Only do the most accurate exception capture

We sometimes want to make script work the king’s way, so no matter what, we will make a big try… Except to wrap the whole code, but it’s easy to swallow the attibuteerror that should have been thrown. Thus, it adds unnecessary trouble to our debugging process.

Therefore, we always capture only those statement blocks that may throw exceptions, and try to capture only the exact exception type, rather than the fuzzy exception.

from requests.exceptions import RequestException

def save_website_title(url, filename):
    try:
        resp = requests.get(url)
    except RequestException as e:
        print(f'save failed: unable to get page content: {e}')
        return False
        
#This regular operation itself should not throw an exception, so we don't need to use the try statement block
#If the group is mistyped as a group, it doesn't matter. The program will come through attributeerror immediately
#Tell us.
    obj = re.search(r'<title>(.*)</title>', resp.text)
    if not obj:
    print('save failed: title tag not found in page content')
    return False
    title = obj.group(1)

    try:
    with open(filename, 'w') as fp:
        fp.write(title)
    except IOError as e:
    print(f'save failed: unable to write to file {filename}: {e}')
    return False
    else:
    return True

3) Exception handling shouldn’t be noisy

As mentioned in the previous article, exception capture should be accurate, but if each one is accurate, there will be many try… Except statement blocks in our code, so as to disturb the core code and the overall readability of the code.

Here, we can use context manager to improve our exception handling process and simplify duplicate exception handling logic.

class raise_api_error:
    """captures specified exception and raise ApiErrorCode instead
    :raises: AttributeError if code_name is not valid
    """
    def __init__(self, captures, code_name):
    self.captures = captures
    self.code = getattr(error_codes, code_name)

    def __enter__(self):
    #This method will be called when entering the context
    return self

    def __exit__(self, exc_type, exc_val, exc_tb):
    #This method will be called when the context exits
    #Exc_type, exc_val, exc_tb represent the
    #Exception type, exception value, error stack
    if exc_type is None:
        return False

    if exc_type == self.captures:
        raise self.code from exc_val
    return False

In the above code, we define a context manager named raise_api_error, which does nothing when entering the context. However, when exiting the context, it will determine whether an exception of type self.features has been thrown in the current context. If so, it will be replaced by the apierrorcode exception class.

After using the context manager, the concise code is as follows:

def upload_avatar(request):
    "" "users upload new avatars" ""
    with raise_api_error(KeyError, 'AVATAR_FILE_NOT_PROVIDED'):
    avatar_file = request.FILES['avatar']
        
    with raise_api_error(ResizeAvatarError, 'AVATAR_FILE_INVALID'),\
        raise_api_error(FileTooLargeError, 'AVATAR_FILE_TOO_LARGE'):
      resized_avatar_file = resize_avatar(avatar_file)

    with raise_api_error(Exception, 'INTERNAL_SERVER_ERROR'):
    request.user.avatar = resized_avatar_file
    request.user.save()
    return HttpResponse({})

πŸ“– Reference

  • Fun Python library tqdm
  • Errors and Exceptions
  • Master Python exception handling, and this article is right for the actual battle of| goose factory
  • Don’t know how to use Python exceptions?

Recommended Today

PHP realizes UnionPay business H5 payment

UnionPay business H5 payment interface document: document address 1: H5 payment interface address: 1: Alipay payment Test address: http://58.247.0.18:29015/v1/netpay/trade/h5-pay Official address: https://api-mop.chinaums.com/ 2: UnionPay payment Test address: http://58.247.0.18:29015/v1/netpay/uac/order Official address: https://api-mop.chinaums.com/ 2: Basic parameters required by the interface The interface uses get parameters. After the interface parameters are directly put into the interface address, the […]