And also, you should also control the memory and CPU usage, as it can point you towards new portions of code that could be improved. Therefore, in this post I’ll comment on 7 different Python tools that give you some insight about the execution time of your functions and the Memory and CPU usage. Use a decorator to time your functions. An OS-specific virtual memory manager carves out a chunk of memory for the Python process. The darker gray boxes in the image below are now owned by the Python process. Python uses a portion of the memory for internal use and non-object memory. The other portion is dedicated to object storage (your int, dict, and the like). Note that this was.

• Python List Memory
• Python Slots Memory Game

Python has two similar sequence types such as tuples and lists. The most well-known difference between them is that tuples are immutable, that is, you cannot change their size as well as their immutable objects.

You can't changes items in a tuple:

But you can change mutable objects:

Internally, both lists and tuples are implemented as a list of pointers to the Python objects (items). When you remove an item from a list, the reference to an item gets destroyed. Keep in mind, that removed item can stay alive if there are other references in your program to it.

Tuples

Despite the fact that tuples are less popular than lists, it is a fundamental data type, which is used a lot internally.

You may not notice, but you are using tuples when:

• working with arguments and parameters
• returning 2 or more items from a function
• iterating over dictionary's key-value pairs
• using string formatting

Typically, a running program has thousands of allocated tuples.

Empty lists vs. empty tuples

Empty tuple acts as a singleton, that is, there is always only one tuple with a length of zero. When creating an empty tuple Python points to already preallocated one, in such way that any empty tuple has the same address in the memory. This is possible because tuples are immutable and sometimes saves a lot of memory.

But this doesn't apply to lists since they can be modified.

Allocation optimization for small tuples

To reduce memory fragmentation and speed up allocations, Python reuses old tuples. If a tuple no longer needed and has less than 20 items instead of deleting it permanently Python moves it to a free list.

A free list is divided into 20 groups, where each group represents a list of tuples of length n between 0 and 20. Each group can store up to 2 000 tuples. The first (zero) group contains only 1 element and represents an empty tuple.

In the example above we can see that a and b have the same id. That is because we immediately occupied a destroyed tuple which was on the free list.

Allocation optimization for lists

Since lists can be modified, Python does not use the same optimization as in tuples. However, Python lists also have a free list, but it is used only for empty objects. If an empty list is deleted or collected by GC, it can be reused later.

List resizing

To avoid the cost of resizing, Python does not resize a list every time you need to add or remove an item. Instead, every list has a number of empty slots which are hidden from a user but can be used for new items. If the slots are completely consumed Python over-allocates additional space for them. The number of additional slots is chosen based on the current size of the list.

Developer documentation describes it as follows:

This over-allocates proportional to the list size, making room for additional growth. The over-allocation is mild but is enough to give linear-time amortized behavior over a long sequence of appends() in the presence of a poorly-performing system realloc().

The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...

Note: new_allocated won't overflow because the largest possible value is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.

For example, if you want to append an item to a list of length 8, Python will resize it to16 slots and add the 9th item. The rest of the slots will be hidden and reserved for new items.

The growing factor looks as follows:

Performance

If you are interested in speed comparison, there is a good summary about the overall performance by Raymond Hettinger.

Want a monthly digest of these blog posts?

• AutoSniper 2 years, 6 months ago (from disqus) #

  This was educational. I encourage using generators and lazy evaluation whenever possible; it is preferred over working with tuples and lists.

Python List Memory

Originally posted on towardsdatascience.

A simple way to improve your Python code

When we create an object from a class, the attributes of the object will be stored in a dictionary called __dict__. We use this dictionary to get and set attributes. It allows us to dynamically create new attributes after the creation of the object.

Let’s create a simple class Article that initially has 2 attributes date and writer. If we print out __dict__ of the object, we will get the key and value of each attribute. Meanwhile, we also print __dict__ of the class which will be needed later. After that, a new attribute reviewer is added to the object, and we can see it in the updated __dict__.

Good enough?

Well, we can’t say this is bad until we find a better solution. Dictionary is very powerful in Python, but when it comes to creating thousands or millions of objects, we might face some issues:

1. Dictionary needs memory. Millions of objects will definitely eat up the RAM usage.
2. Dictionary is in fact a hash map. The worst case of the time complexity of get/set in a hash mapis O(n).

__slots__ solution

From Python documentation: __slots__ allows us to explicitly declare data members (like properties) and deny the creation of __dict__ and __weakref__ (unless explicitly declared in __slots__ or available in a parent.)

So, how does it relate to the issues I’ve mentioned?

Let’s create a class ArticleWithSlots. The only difference between 2 classes is the extra field __slots__.

__slots__ is created on the class level, which means if we print ArticleWithSlots.__dict__, we should be able to see it. Besides, we also see 2 extra attributes on the class level, date: <member 'date' ..> and writer: <member 'writer' ..>, which belong to class member_descriptor.

What is a descriptor in Python?

Before we talk about descriptor, we should understand the default behaviour of accessing attributes in Python. When you do article.writer, Python will call the method __getattribute__(), where it does a look up in __dict__, self.__dict__['writer'] and return the value.

If the look up key is an object with one of the descriptor methods, then the default behaviour will be overwritten by the descriptor method.

Descriptor methods include __get__(), __set__() and __delete__(). And a descriptor is simply a Python object that implements at least one descriptor methods.

__slots__ automatically creates a descriptor for each attribute with the implementation of descriptor methods. You can find them in the screenshot. It means that the object will use __get__(), __set__() and __delete__() to interact with attributes instead of the default behavior.

Python Slots Memory Game

According to Guido van Rossum, the implementation of __get__(), __set__() uses an array instead of the dictionary and it’s entirely implemented in C which is highly efficient.

__slots__ has faster attribute access

In the following code, I compare the object creation time and attribute access time of Article and ArticleWithSlots. __slots__ is around 10% faster.

__slots__ has slightly better performance is because thetime complexity of get/set operationin a list is faster than a dictionary in the worst case. As O(n) only happens in the worst case, we will not notice the difference most of the time, especially when you have a small volume of data.

__slots__ reduces RAM usage

Since attributes can be accessed as data members (like properties), there is no need to store them in the dictionary __dict__. Actually, __slots__ will deny the creation of __dict__ at all. So if you print article_slots.__dict__, you will get the AttributeError exception.

And this behavior reduces the RAM usage of an object. I will compare the size of article and article_slots using pympler. The reason for not using sys.getsizeof() is that getsizeof() doesn’t include the size of referenced objects. However, __dict__ is a referenced object which will be ignored in getsizeof().

It turns out that article_slotssaves more than 50% of the memory. WoW, such an amazing improvement!

Such a good performance is because article_slots doesn’t have __dict__ attribute which actually saves a lot of memory.

When to use and not use __slots__?

So far, it looks like __slots__ is such a nice feature. Can we add it to every class?

The answer is NO! Apparently, there are some trade-offs.

Fixed attributes

One of the reasons to use __dict__ is its flexibility after creating the object where you can add new attributes. However, __slots__ will fix the attributes when you create the class. So, it’s not possible to add new attributes later.

But …

In some cases where you want to take advantage of __slots__, and also have the flexibility of adding new attributes in the runtime. You can achieve this by adding __dict__ inside __slots__ as an attribute. Only the newly added attributes will appear in __dict__. This can be useful when your class has 10+ fixed attributes and you want to have 1 or 2 dynamic attributes later.

Inheritance

If you want to inherit a class that includes __slots__, you don’t have to repeat those attributes again in the subclass. Otherwise, the subclass will take up more space. Besides, the repeated attributes will be inaccessible in the parent class.

It works the same when you inherit a NamedTuple. You don’t need to repeat attributes in the subclass. If you want to understand more about NamedTuple, you can read my article dedicated to this topic.

You can also add __dict__ attribute in the subclass. Or, you don’t put __slots__ in the subclass, it will by default have __dict__.

If you inherit a class without __slots__, then the subclass will contain __dict__.

Conclusion

I hope you’ve understood what __slots__ is and some details of the implementation. At the end of the article, I want to share the pros and cons that coming from my own experience and the internet (linked in the Reference).

pros

__slots__ can be definitely useful when you have a pressure on memory usage. It’s extremely easy to add or remove with just one-line of code. The possibility of having __dict__ as an attribute in __slots__ gives developers more flexibility to manage attributes while taking care of the performance.

cons

You need to be clear about what you are doing and what you want to achieve with __slots__, especially when inheriting a class with it. The order of inheritance, the attribute names can make a huge difference in the performance.

You can’t inherit a built-in type such as int, bytes, tuple with non-empty __slots__. Besides, you can’t assign a default value to attributes in __slots__. This is because these attributes are supposed to be descriptors. Instead, you can assign the default value in __init__().

Source: towardsdatascience