Lists, tuples and dictionaries

Lesson 2 — Python

Lesson 2 of 20 Beginner ~80 min

Learning objectives

Use lists, tuples, dictionaries and sets, and choose between them
Write comprehensions for filtering and transforming
Explain mutability and why it causes surprising bugs
Copy collections correctly, shallow and deep
Use the collections module for common patterns

Lists

Ordered, mutable, any types mixed.

x = [1, 2, 3]
mixed = [1, "a", 3.14, None, [1, 2]]
empty = []

x[0]            # 1
x[-1]           # 3
x[1:]           # [2, 3]
len(x)          # 3

x.append(4)             # [1, 2, 3, 4]        add one
x.extend([5, 6])        # [1, 2, 3, 4, 5, 6]  add many
x.insert(0, 0)          # [0, 1, 2, ...]      at a position
x.remove(3)             # remove first occurrence of the VALUE 3
x.pop()                 # remove and return the last
x.pop(0)                # remove and return position 0
x.sort()                # sorts IN PLACE, returns None
x.reverse()             # in place
x.count(2)
x.index(2)              # position of the first 2
x.clear()

sorted(x)               # returns a NEW sorted list
sorted(x, reverse=True)
sorted(people, key=lambda p: p["age"])
reversed(x)             # an iterator

3 in x                  # membership
x + [7, 8]              # concatenation, new list
x * 2                   # repetition

In-place versus returning

x = [3, 1, 2]
y = x.sort()        # y is None! sort() mutates x and returns nothing
x                   # [1, 2, 3]

y = sorted(x)       # y is a new sorted list, x unchanged

This trips up every R user. list.sort(), .append(), .reverse() and .extend() all mutate and return None. The functions sorted() and reversed() return new objects.

Tuples

Ordered, immutable.

t = (1, 2, 3)
t = 1, 2, 3             # parentheses are optional
single = (1,)           # the trailing comma is what makes it a tuple
empty = ()

t[0]                    # 1
len(t)                  # 3
t[0] = 99               # TypeError: 'tuple' object does not support item assignment

Tuples are used for:

# Multiple return values
def min_max(x):
    return min(x), max(x)

lo, hi = min_max([3, 1, 4])       # unpacking

# Swapping
a, b = b, a

# Unpacking with a rest
first, *rest = [1, 2, 3, 4]       # first=1, rest=[2,3,4]
first, *middle, last = [1, 2, 3, 4]

# Dictionary keys (lists cannot be keys — they are mutable)
lookup = {(2026, 3): "March 2026"}

# Fixed records
point = (10, 20)

Use a tuple when the collection has a fixed size and meaning; a list when it is a variable-length collection of similar things.

Dictionaries

Key-value mapping. Ordered by insertion since Python 3.7.

d = {"name": "Ram", "age": 45}
d = dict(name="Ram", age=45)
empty = {}

d["name"]               # "Ram"
d["missing"]            # KeyError!
d.get("missing")        # None       — safe
d.get("missing", 0)     # 0          — with a default

d["email"] = "a@b.com"  # add or update
del d["email"]
d.pop("age")            # remove and return
d.pop("age", None)      # with a default, no KeyError

"name" in d             # True — checks KEYS
d.keys()
d.values()
d.items()               # (key, value) pairs

d.update({"age": 46, "city": "Hyderabad"})
d.setdefault("count", 0)     # set only if absent

{**d1, **d2}            # merge, d2 wins
d1 | d2                 # same, Python 3.9+

Iteration:

for key in d:                       # keys by default
    print(key)

for key, value in d.items():
    print(f"{key}: {value}")

for value in d.values():
    print(value)

Dictionaries are the Python equivalent of R’s named list, and they are used everywhere — JSON, function keyword arguments, configuration, DataFrame construction.

Sets

Unordered, unique, fast membership testing.

s = {1, 2, 3}
s = set([1, 2, 2, 3])       # {1, 2, 3}
empty = set()               # NOT {} — that is an empty dict

s.add(4)
s.discard(2)                # no error if absent
s.remove(2)                 # KeyError if absent

a = {1, 2, 3}
b = {2, 3, 4}
a | b                       # union         {1,2,3,4}
a & b                       # intersection  {2,3}
a - b                       # difference    {1}
a ^ b                       # symmetric difference {1,4}
a <= b                      # subset

Sets are the right tool for deduplication and membership:

# Which subjects are in ADAE but not ADSL?
set(adae_ids) - set(adsl_ids)

# Fast lookup — O(1) versus O(n) for a list
valid = set(adsl_ids)
[x for x in records if x.usubjid in valid]

That last pattern matters: x in some_list scans the list, so a loop over 100,000 records checking membership in a 50,000-element list is 5 billion comparisons. Converting to a set makes it instant.

Comprehensions

The most characteristic Python idiom. They replace sapply, Filter and Map.

# Map
squares = [x**2 for x in range(10)]

# Filter
evens = [x for x in range(20) if x % 2 == 0]

# Both
result = [x**2 for x in range(20) if x % 2 == 0]

# Transform with a conditional
labels = ["high" if x > 5 else "low" for x in values]

# Nested
pairs = [(a, b) for a in [1, 2] for b in ["x", "y"]]
#> [(1,'x'), (1,'y'), (2,'x'), (2,'y')]

# Flatten
flat = [x for sublist in nested for x in sublist]

# Dictionary comprehension
lengths = {word: len(word) for word in words}
inverted = {v: k for k, v in d.items()}

# Set comprehension
unique_first = {name[0] for name in names}

# Generator expression — lazy, no parentheses needed inside a call
total = sum(x**2 for x in range(1_000_000))

Generator versus list comprehension

[x**2 for x in range(10_000_000)]     # builds a 10M-element list in memory
(x**2 for x in range(10_000_000))     # a generator; computes on demand

Use a generator when you are consuming the values once — sum(), max(), any(), a for loop. Use a list when you need to index into it or iterate more than once.

Readability limit: if a comprehension needs more than one for and one if, write a loop.

# Too clever
result = [f(x) for sub in data for x in sub if g(x) and h(x, sub)]

# Clearer
result = []
for sub in data:
    for x in sub:
        if g(x) and h(x, sub):
            result.append(f(x))

Mutability

The concept that causes the most surprising Python bugs.

a = [1, 2, 3]
b = a               # NOT a copy — both names point to the SAME list
b.append(4)
a                   # [1, 2, 3, 4]   — a changed too!

# R copies on modify
a <- c(1, 2, 3)
b <- a
b[4] <- 4
a                   # 1 2 3 — unchanged

This is the deepest difference between the languages, and it applies to lists, dictionaries and sets — but not to strings, tuples, ints or floats, which are immutable.

Copying

import copy

a = [1, 2, 3]
b = a.copy()            # shallow copy
b = a[:]                # same
b = list(a)             # same

nested = [[1, 2], [3, 4]]
shallow = nested.copy()
shallow[0].append(99)
nested                  # [[1, 2, 99], [3, 4]]   — inner lists are shared!

deep = copy.deepcopy(nested)
deep[0].append(99)
nested                  # unchanged

A shallow copy copies the outer container but not the objects inside it. For a list of lists, a list of dicts, or a DataFrame containing objects, you usually want deepcopy.

Mutable default arguments

The classic Python trap:

def add_item(item, items=[]):        # WRONG
    items.append(item)
    return items

add_item("a")        # ["a"]
add_item("b")        # ["a", "b"]    — the default persists between calls!

The default is evaluated once, when the function is defined. The fix:

def add_item(item, items=None):
    if items is None:
        items = []
    items.append(item)
    return items

This is important enough that linters flag it, and you should never use a mutable default.

Mutation in loops

values = [1, 2, 3, 4, 5]
for v in values:
    if v % 2 == 0:
        values.remove(v)      # modifying while iterating
values
# [1, 3, 5]  — looks right, but it is luck

Removing while iterating skips elements. Build a new list instead:

values = [v for v in values if v % 2 != 0]

The `collections` module

from collections import Counter, defaultdict, namedtuple, deque

# Counter — frequency counting
c = Counter(["a", "b", "a", "c", "a"])
c["a"]                  # 3
c.most_common(2)        # [("a", 3), ("b", 1)]
c.total()               # 5

# defaultdict — no KeyError, creates on access
by_subject = defaultdict(list)
for record in records:
    by_subject[record["usubjid"]].append(record)
# no need to check whether the key exists first

counts = defaultdict(int)
for term in terms:
    counts[term] += 1

# namedtuple — a lightweight record with named fields
Subject = namedtuple("Subject", ["id", "age", "arm"])
s = Subject("001", 45, "Placebo")
s.age                   # 45
s[1]                    # 45 — still indexable

# deque — fast append and pop at BOTH ends
d = deque([1, 2, 3])
d.appendleft(0)
d.popleft()

defaultdict(list) is the Python answer to “group these records by a key” without pandas, and it appears constantly in data processing code.

R and Python side by side

Task	R	Python
Ordered collection	`c(1, 2, 3)` or `list()`	`[1, 2, 3]`
Named collection	`list(a = 1, b = 2)`	`{"a": 1, "b": 2}`
Get element	`x[[1]]`, `x$a`	`x[0]`, `x["a"]`
Safe get	`x[["a"]]` (NULL if absent)	`x.get("a")`
Length	`length(x)`	`len(x)`
Append	`c(x, 4)`	`x.append(4)`
Map	`sapply(x, f)`	`[f(i) for i in x]`
Filter	`Filter(f, x)`	`[i for i in x if f(i)]`
Names	`names(x)`	`x.keys()`
Unique	`unique(x)`	`set(x)` or `list(dict.fromkeys(x))`
Membership	`3 %in% x`	`3 in x`
Set difference	`setdiff(a, b)`	`set(a) - set(b)`
Sort	`sort(x)`	`sorted(x)`
Assignment copies	Yes	No

The last row is the one to internalise.

Common mistakes

Mistake	Consequence	Fix
`b = a` expecting a copy	Both change	`a.copy()` or `copy.deepcopy()`
`x = list.sort()`	`x` is `None`	`x = sorted(list)`
Mutable default argument	State leaks between calls	Default `None`
Shallow copy of nested data	Inner objects shared	`copy.deepcopy()`
`d["key"]` on a missing key	`KeyError`	`d.get("key", default)`
Membership test against a big list	Very slow	Convert to a `set`
Modifying a list while iterating	Skipped elements	Build a new list
`{}` for an empty set	It is a dict	`set()`

Exercise 2.1 — Group records by key

Given a list of dictionaries representing adverse events, write a function that groups them by usubjid and returns, for each subject, the count of events and the set of distinct preferred terms.

events = [
    {"usubjid": "001", "term": "Headache", "severity": "MILD"},
    {"usubjid": "001", "term": "Nausea",   "severity": "MODERATE"},
    {"usubjid": "001", "term": "Headache", "severity": "MILD"},
    {"usubjid": "002", "term": "Rash",     "severity": "SEVERE"},
]

Show solution

from collections import defaultdict


def summarise_by_subject(events):
    """Summarise adverse events per subject.

    Returns a dict keyed by usubjid, each value a dict with the event count,
    the distinct preferred terms, and the worst severity.
    """
    SEVERITY_RANK = {"MILD": 1, "MODERATE": 2, "SEVERE": 3}

    grouped = defaultdict(list)
    for e in events:
        grouped[e["usubjid"]].append(e)

    return {
        subj: {
            "n_events": len(evs),
            "terms": sorted({e["term"] for e in evs}),
            "n_distinct_terms": len({e["term"] for e in evs}),
            "worst_severity": max(
                (e["severity"] for e in evs),
                key=lambda s: SEVERITY_RANK.get(s, 0),
            ),
        }
        for subj, evs in grouped.items()
    }


import json
print(json.dumps(summarise_by_subject(events), indent=2))

{
  "001": {
    "n_events": 3,
    "terms": ["Headache", "Nausea"],
    "n_distinct_terms": 2,
    "worst_severity": "MODERATE"
  },
  "002": {
    "n_events": 1,
    "terms": ["Rash"],
    "n_distinct_terms": 1,
    "worst_severity": "SEVERE"
  }
}

The pieces:

defaultdict(list) avoids if subj not in grouped: grouped[subj] = [] on every iteration. This is the canonical grouping idiom in plain Python.
A set comprehension {e["term"] for e in evs} deduplicates the terms; sorted() makes the output deterministic, which matters if this feeds a test.
max(..., key=...) with a rank dictionary finds the worst severity by an ordering that is not alphabetical. max() on the raw strings would give "SEVERE" by luck here but "MODERATE" over "MILD" correctly and "MODERATE" over "SEVERE" incorrectly in general — "S" > "M" is true, so it happens to work, which is exactly the kind of accident not to rely on.

With pandas this is three lines — see lesson 5 — but the plain Python version is worth being able to write, because a great deal of code processes JSON and API responses that are never DataFrames.

Exercise 2.2 — Reconcile two datasets

Given lists of subject IDs from ADSL and ADAE, report: subjects in both, subjects only in ADSL, subjects only in ADAE, and whether either list has duplicates. Explain why a set is the right tool.

Show solution

from collections import Counter


def reconcile(adsl_ids, adae_ids):
    """Compare subject identifiers between two datasets."""
    adsl_set = set(adsl_ids)
    adae_set = set(adae_ids)

    adsl_dupes = {k: v for k, v in Counter(adsl_ids).items() if v > 1}
    adae_dupes = {k: v for k, v in Counter(adae_ids).items() if v > 1}

    return {
        "n_adsl": len(adsl_ids),
        "n_adsl_distinct": len(adsl_set),
        "n_adae": len(adae_ids),
        "n_adae_distinct": len(adae_set),
        "in_both": sorted(adsl_set & adae_set),
        "adsl_only": sorted(adsl_set - adae_set),
        "adae_only": sorted(adae_set - adsl_set),
        "adsl_duplicates": adsl_dupes,
        "adae_duplicates": adae_dupes,
    }


adsl = ["001", "002", "003", "004"]
adae = ["002", "003", "003", "005"]

result = reconcile(adsl, adae)
for k, v in result.items():
    print(f"{k:20} {v}")

n_adsl               4
n_adsl_distinct      4
n_adae               4
n_adae_distinct      3
in_both              ['002', '003']
adsl_only            ['001', '004']
adae_only            ['005']
adsl_duplicates      {}
adae_duplicates      {'003': 2}

Why a set

Three reasons:

The operations exist. &, - and | express intersection, difference and union directly. The list equivalent is a nested loop or a comprehension with a membership test, and it is both slower and harder to read.
Membership is O(1). A set is a hash table. Checking x in some_set takes constant time regardless of size; x in some_list scans. With 50,000 subjects on each side, the list version is ~2.5 billion comparisons and takes minutes; the set version is instantaneous.

import timeit

big_list = list(range(100_000))
big_set = set(big_list)

timeit.timeit(lambda: 99_999 in big_list, number=1000)   # ~0.9 s
timeit.timeit(lambda: 99_999 in big_set,  number=1000)   # ~0.00005 s

Deduplication is implicit, which is what you want for “which subjects appear in both” — but not what you want for detecting duplicates. Hence Counter for that part.

That last point is the design lesson: a set answers “which values” and a Counter answers “how many times”. Using a set alone would silently hide the duplicate "003" in ADAE, which is exactly the data problem you most want to find.

The adae_only result is the important one. A subject with adverse events who is not in ADSL means either a data error or a population definition mismatch, and it should stop the pipeline — the same check as relationship = "many-to-one" in R’s dplyr joins.

Recap

Lists are mutable and ordered; tuples immutable; dicts are key-value; sets are unique
b = a does not copy — this is the deepest difference from R
.sort() mutates and returns None; sorted() returns a new list
Comprehensions replace sapply/Filter/Map; generators for one-pass consumption
Never use a mutable default argument
defaultdict(list) for grouping, Counter for frequencies
Convert to a set before repeated membership tests

Next: Functions and classes.