Sept. 8, 2020

Count number of elements, and memory allocated

Length provides a basic count of first level elements with list. Each flat element takes 8 bytes to store and memory is mostly allocated linearly for lists. The structure of a list is determined recursively by nestedness and type of objects in the container.


Page contents

Primary length as structure metrics

Basic count of elements

Py3: Elements within container

#   simple count of items
lst_a = [1, 2, 3]
k = len(lst_a)
print('Length =',k)     #= 3

#   nested list
lst_a = [1, 2, [3, 4, 5]]
k = len(lst_a)
print('Length =',k)     #= 3

#   empty elements
lst_a = ['', [], {}]
k = len(lst_a)
print('Length =',k)     #= 3

Notes: Len method

The length method counts the number of first level elements within the container. It does not discriminate if elements are nested or empty.

Nested structure summary of a list

Analyze a nested list for elements.

Py3: Structure summary

#   summarize a nested iterable
#   count nesting level, all elements
#   distinguish between empty and other true elements
def structure(tgt):
    struct = {}
    empty = {}
    level = 1
    #   make a copy to prevent changing original
    obj = tgt.copy()
    while True:
        rem = []
        for item in obj:
            #   nested item
            if isinstance(item, (list, tuple, set)):
                rem.extend(item)

            #   level and object
            key = (level, type(item))
            if key in struct:
                struct[key] += 1
            else:
                struct[key] = 1

            #   empty containers
            if not item:
                if key in empty:
                    empty[key] += 1
                else:
                    empty[key] = 1

        if not rem:
            #   if no more nesting
            break
        else:
            #   parse next level
            obj = rem
            level += 1

    #   summarize results
    sum_type = {}
    sum_level = {}
    for k, v in struct.items():
        lvl, typ = k
        #   level
        if lvl in sum_level:
            sum_level[lvl] += v
        else:
            sum_level[lvl] = v
        #   type
        if typ in sum_type:
            sum_type[typ] += v
        else:
            sum_type[typ] = v

    #   raw structure, empty items, summaries
    return (struct, empty, sum_type, sum_level)


#   sample to summarize
nested = [1,'abc',(3,),[4,(5,6),[],'', [8,'n','e','s'], ['t',[]]]]
s, e, t, v = structure(nested)

print('Length:', len(nested))
print('Structure:', s)
print('Summary type:', t)
print('Summary level:', v)
print('Empty:', e)

#= Length: 4
# Structure: {
#    (1, <class 'int'>): 1,
#    (1, <class 'str'>): 1,
#    (1, <class 'tuple'>): 1,
#    (1, <class 'list'>): 1,
#    (2, <class 'int'>): 2,
#    (2, <class 'tuple'>): 1,
#    (2, <class 'list'>): 3,
#    (2, <class 'str'>): 1,
#    (3, <class 'int'>): 3,
#    (3, <class 'str'>): 4,
#    (3, <class 'list'>): 1
# }
# Summary type: {
#    <class 'int'>: 6,
#    <class 'str'>: 6,
#    <class 'tuple'>: 2,
#    <class 'list'>: 5
# }
# Summary level: {
#    1: 4,
#    2: 7,
#    3: 8
# }
# Empty: {
#    (2, <class 'list'>): 1,
#    (2, <class 'str'>): 1,
#    (3, <class 'list'>): 1
# }


# for an empty list
#------------------
nested = []

#= Length: 0
# Structure: {}
# Summary type: {}
# Summary level: {}
# Empty: {}


# for a nested empty list
#--------------
nested = [[[]]]

#= Length: 1
# Structure: {(1, <class 'list'>): 1, (2, <class 'list'>): 1}
# Summary type: {<class 'list'>: 2}
# Summary level: {1: 1, 2: 1}
# Empty: {(2, <class 'list'>): 1}

Notes: Detailed structure of nested lists

The structure function analyzes a nested list for degree of nestedness. It summarizes all elements, and also by nested level. Empty and non-empty elements are counted and summarized separately. Function returns the type of object present within each level. The function traverses the list structure recursively making dictionaries of object types and counts.

Memory allocation using sizeof

Check size of object memory allocation.

Py3: Memory allocation

#   empty list requires 40 bytes
k = list()
print(k.__sizeof__())   #= 40

#   single element requires extra 32 bytes
k = [1]
print(k.__sizeof__())   #= 72
Element Count List Memory Tuple Memory Set Memory
0 40, 32 24, 8 200, 0
1 72, 8 32, 8 200, 0
2 80, 8 40, 8 200, 0
3 88, 8 48, 8 200, 0
4 96, 8 56, 8 200, 512
5 104, 8 64, 8 712, 0
6 112, 8 72, 8 712, 0
7 120, 16 80, 8 712, 0
8 136, 32 88, 8 712, 0
9 168, 8 96, 8 712, 0
10 176, 8 104, 8 712, 0
11 184, 8 112, 8 712, 0
12 192, 8 120, 8 712, 0
13 200, 8 128, 8 712, 0
14 208, 8 136, 8 712, 0
15 216, 16 144, 8 712, 0
16 232, 8 152, 8 712, 0
17 240, 8 160, 8 712, 0
18 248, 8 168, 8 712, 1536
19 256, 8 176, 8 2248, 0
20 264, 8 184, 8 2248, 0
21 272, 8 192, 8 2248, 0
22 280, 8 200, 8 2248, 0
23 288, 16 208, 8 2248, 0

Notes: Memory allocated for different containers

An empty list takes up 40 bytes, which is slightly more than tuple but much less than a set. Adding a single element adds 32 bytes. For more elements, the increase is generally 8 bytes, with a few 16 byte increments. A tuple increase linearly, and sets allocate a lot of memory at certain discrete element counts.

Popular content

python programming

Read and write lists with Pickle, Array and Pandas

python programming

Flatten nested list or generate blocks of nested lists

python programming

For loop and control statements

python programming

Clear list using inplace and standard methods

python programming

List comprehension with nested conditions

python programming

Concatenate list elements using add, append, extend

python programming

Enumerate and custom counters like skip and loop

python programming

Count number of elements, and memory allocated

python programming

Remove duplicate list elements

python programming

Statistics with numeric lists of integers, fractions, and decimals.

New content

python programming

Read and write lists with Pickle, Array and Pandas

python programming

Is element in list?

python programming

Dictionary merge common key groups

python programming

Packaging loops with zip

python programming

Concatenate list elements using add, append, extend

python programming

List comprehension with nested conditions

python programming

Flatten nested list or generate blocks of nested lists

python programming

Enumerate and custom counters like skip and loop

python programming

Range integer sequences

python programming

For loop and control statements