Week 2: Saving data and collecting it every day

Goal of today: every morning, without you doing anything, tomorrow's forecasts of the 3 services for both cities are saved into a database file, weather.db:

2026-10-06 08:00:01,336 budapest  open_meteo  2026-10-07 max 25.0 min 12.3
2026-10-06 08:00:01,467 budapest  met_norway  2026-10-07 max 22.8 min 9.0
2026-10-06 08:00:01,570 budapest  wttr        2026-10-07 max 25.0 min 14.0
2026-10-06 08:00:01,649 eger      open_meteo  2026-10-07 max 24.4 min 11.9
2026-10-06 08:00:01,783 eger      met_norway  2026-10-07 max 22.0 min 10.4
2026-10-06 08:00:01,886 eger      wttr        2026-10-07 max 23.0 min 10.0

Start the daily collection this week, not later. Machine learning needs data, and your own data grows only one day per day. What you collect from now on is what you will use in week 8.

How to work: do the steps in order. After each step, run the code and compare with the expected output. (Your temperatures and times will differ.)

What you need to know

Database, table, row, column. A database stores data in tables. A table is like a sheet in Excel: every row is one record (here: one forecast), every column is one property of it (city, day, max temperature, ...).

Here are two example rows of our forecasts table. The real table gets 6 new rows every day (3 services × 2 cities), so it will have hundreds of rows by the end of the semester:

city	source	day	tmax	tmin	saved_at
eger	wttr	2026-10-07	23.0	10.0	2026-10-06T08:00:01
eger	met_norway	2026-10-07	22.0	10.4	2026-10-06T08:00:01

SQLite is a database in a single file. It is built into Python (import sqlite3), there is nothing to install.

SQL is the language you talk to a database in. You need only three commands:

SQL	Meaning
`CREATE TABLE temps (city TEXT, day TEXT, tmax REAL)`	make a new table with 3 columns (`TEXT` = text, `REAL` = decimal number)
`INSERT INTO temps VALUES ('eger', '2026-10-07', 23.0)`	add a row
`SELECT * FROM temps WHERE city = 'eger'`	read rows (`*` = all columns, `WHERE` = only the matching rows)

Primary key. PRIMARY KEY (city, day) means: there can be only one row with the same city and day. Inserting a second one is an error. INSERT OR REPLACE instead overwrites the old row. This protects us from duplicates if the program runs twice on the same day.

? placeholders. In Python you do not write the values into the SQL text. You write ? and give the values separately: con.execute("SELECT * FROM temps WHERE city = ?", ("eger",)). This is safer, and quotes or numbers cannot break the SQL.

Commit. Changes are really written into the file only after con.commit(). Without it, they are lost.

Logging. A program started automatically at 08:00 has no window you could look at. So it writes what it did into a text file, a log. If something fails, you read it there later.

Scheduled task. Windows can start a program at a given time every day: this is the Task Scheduler. We use it to run our collector every morning.

Step 1: `config.py`: where the database file is

Open config.py from week 1 and change it to this (new: the Path import, BASE_DIR, DB_PATH and PROVIDER_SOURCES):

config.py

"""Settings shared by all scripts."""
from pathlib import Path

# Files are always created next to this file, no matter where you start Python from.
BASE_DIR = Path(__file__).resolve().parent
DB_PATH = BASE_DIR / "weather.db"

TIMEZONE = "Europe/Budapest"

CITIES = {
    "budapest": {"name": "Budapest", "lat": 47.4979, "lon": 19.0402},
    "eger": {"name": "Eger", "lat": 47.9025, "lon": 20.3772},
}

# MET Norway wants to know who is asking. You may add your own e-mail address,
# but NOT a fake one like ...@example.com (that is blocked with error 403).
USER_AGENT = "weather-lab/1.0 (university lab project)"

# Live set: three independent forecast providers, collected by us every day.
PROVIDER_SOURCES = ["open_meteo", "met_norway", "wttr"]

Path(__file__) is the path of config.py itself; .parent is its folder. So BASE_DIR is your project folder.
DB_PATH is the database file in the project folder. This matters later: Task Scheduler starts the program from another folder, and without BASE_DIR the database would be created there.
PROVIDER_SOURCES is the list of our three services. We need it from week 3.

Try it:

python
>>> from config import DB_PATH, PROVIDER_SOURCES
>>> DB_PATH
WindowsPath('C:/weather-lab/weather.db')
>>> PROVIDER_SOURCES
['open_meteo', 'met_norway', 'wttr']
>>> exit()

Step 2: Try SQL by hand

Before writing the real code, try the three SQL commands in the interactive Python. This creates a test file, test.db, which we delete at the end.

python
>>> import sqlite3
>>> con = sqlite3.connect("test.db")
>>> con.execute("CREATE TABLE temps (city TEXT, day TEXT, tmax REAL, PRIMARY KEY (city, day))")
<sqlite3.Cursor object at 0x...>
>>> con.execute("INSERT INTO temps VALUES ('eger', '2026-10-07', 23.0)")
<sqlite3.Cursor object at 0x...>
>>> con.execute("INSERT INTO temps VALUES ('budapest', '2026-10-07', 25.0)")
<sqlite3.Cursor object at 0x...>
>>> con.execute("SELECT * FROM temps").fetchall()
[('eger', '2026-10-07', 23.0), ('budapest', '2026-10-07', 25.0)]

.fetchall() gives back the rows as a list of tuples. (The Cursor object lines are normal, ignore them.)

Now try to insert Eger for the same day again:

>>> con.execute("INSERT INTO temps VALUES ('eger', '2026-10-07', 24.0)")
sqlite3.IntegrityError: UNIQUE constraint failed: temps.city, temps.day

The primary key does not allow a second row for the same city and day. With INSERT OR REPLACE the old row is overwritten:

>>> con.execute("INSERT OR REPLACE INTO temps VALUES ('eger', '2026-10-07', 24.0)")
<sqlite3.Cursor object at 0x...>
>>> con.execute("SELECT * FROM temps").fetchall()
[('budapest', '2026-10-07', 25.0), ('eger', '2026-10-07', 24.0)]

Still two rows, and Eger has the new value 24.0. Finally, a SELECT with a ? placeholder:

>>> con.execute("SELECT tmax FROM temps WHERE city = ?", ("eger",)).fetchall()
[(24.0,)]
>>> con.close()
>>> exit()

(("eger",) with the comma is a tuple with one element. Without the comma it would be just text in brackets.)

Delete test.db now; it was only for trying.

Step 3: `db.py`: opening the database

Create a new file db.py:

"""Everything that touches the SQLite database file (weather.db)."""
import sqlite3
from datetime import datetime

from config import DB_PATH

SCHEMA = """
CREATE TABLE IF NOT EXISTS forecasts (
    city TEXT, source TEXT, day TEXT, tmax REAL, tmin REAL, saved_at TEXT,
    PRIMARY KEY (city, source, day)
);
"""


def connect():
    con = sqlite3.connect(DB_PATH)
    con.executescript(SCHEMA)  # creates the tables the first time, does nothing later
    return con


def now():
    return datetime.now().isoformat(timespec="seconds")

SCHEMA is the SQL that creates our table. IF NOT EXISTS means: create it only the first time.
connect() opens the file (SQLite creates it if it does not exist yet), runs SCHEMA, and returns the connection. So every part of the program can simply call connect() and the table is always there.
The primary key is (city, source, day): one forecast per city, per service, per day.
now() gives the current time as text; we save it with every row, so we know when it was saved.

Try it:

python
>>> from db import connect, now
>>> con = connect()
>>> con.execute("SELECT name FROM sqlite_master WHERE type = 'table'").fetchall()
[('forecasts',)]
>>> con.close()
>>> now()
'2026-10-06T14:34:23'
>>> exit()

sqlite_master is a built-in table that lists all tables. Our forecasts table exists, and weather.db appeared in the folder.

Step 4: `db.py`: saving forecasts

Add this function at the end of db.py:

def save_forecasts(city, source, values):
    """values: {day: (tmax, tmin)}. Saving the same day again overwrites it (no duplicates)."""
    rows = [(city, source, day, tmax, tmin, now()) for day, (tmax, tmin) in values.items()]
    con = connect()
    con.executemany("INSERT OR REPLACE INTO forecasts VALUES (?, ?, ?, ?, ?, ?)", rows)
    con.commit()
    con.close()
    return len(rows)

values is a dictionary like {"2026-10-07": (23.0, 10.0)}: day → (max, min). Today we save one day at a time; in week 3 we will save 900 days with the same function.
The list comprehension turns it into rows: ("eger", "wttr", "2026-10-07", 23.0, 10.0, "2026-10-06T14:34:23").
executemany inserts all rows with one command. The six ? are filled from each row.
It returns how many rows were saved.

Try it: save the same forecast twice, with a different max the second time:

python
>>> from db import save_forecasts, connect
>>> save_forecasts("eger", "wttr", {"2026-10-07": (23.0, 10.0)})
1
>>> save_forecasts("eger", "wttr", {"2026-10-07": (23.5, 10.0)})
1
>>> connect().execute("SELECT * FROM forecasts").fetchall()
[('eger', 'wttr', '2026-10-07', 23.5, 10.0, '2026-10-06T14:34:23')]
>>> exit()

Only one row, with the newer value 23.5. No duplicates.

Delete weather.db now: this test row is made-up data. The next step creates the file again with real data.

Step 5: `collect.py`: the daily job

Create a new file collect.py. It is the loop of first_look.py from week 1, but it saves instead of printing, and it writes a log:

"""Daily job: save tomorrow's forecasts into the database."""
import logging

from config import BASE_DIR, CITIES
from db import save_forecasts
from sources import PROVIDERS, tomorrow

logging.basicConfig(
    level=logging.INFO, format="%(asctime)s %(message)s",
    handlers=[logging.FileHandler(BASE_DIR / "collect.log", encoding="utf-8"), logging.StreamHandler()],
)
log = logging.getLogger()

def collect_forecasts():
    for key, city in CITIES.items():
        # Live set: one provider failing must not stop the others.
        for source, fetch in PROVIDERS.items():
            try:
                tmax, tmin = fetch(city)
                save_forecasts(key, source, {tomorrow(): (tmax, tmin)})
                log.info(f"{key:9s} {source:11s} {tomorrow()} max {tmax} min {tmin}")
            except Exception as error:
                log.error(f"{key:9s} {source:11s} FAILED: {error}")


def run_once():
    collect_forecasts()

if __name__ == "__main__":
    run_once()

logging.basicConfig(...) sets up the log once: every message gets the time in front (%(asctime)s), and goes to two places (handlers): the file collect.log and the screen.
log.info(...) writes a normal message, log.error(...) an error message.
try / except again: if one service fails, we log it and continue with the others.
run_once() looks unnecessary now (it only calls one function), but in week 3 it will do more.
if __name__ == "__main__": means "only when this file is started directly", not when another file imports it.

Run it:

python collect.py

You should see:

2026-10-06 14:34:33,336 budapest  open_meteo  2026-10-07 max 25.0 min 12.3
2026-10-06 14:34:33,467 budapest  met_norway  2026-10-07 max 22.8 min 9.0
2026-10-06 14:34:33,570 budapest  wttr        2026-10-07 max 25.0 min 14.0
2026-10-06 14:34:33,649 eger      open_meteo  2026-10-07 max 24.4 min 11.9
2026-10-06 14:34:33,783 eger      met_norway  2026-10-07 max 22.0 min 10.4
2026-10-06 14:34:33,886 eger      wttr        2026-10-07 max 23.0 min 10.0

Open collect.log in the editor: the same 6 lines are there.

Step 6: Run it again: no duplicates

Run it a second time:

python collect.py

Then count the rows:

python -c "import sqlite3; print(sqlite3.connect('weather.db').execute('SELECT COUNT(*) FROM forecasts').fetchone())"

You should see:

(6,)

Still 6 rows (3 services × 2 cities), not 12: the primary key and INSERT OR REPLACE work. collect.log now has 12 lines: the log keeps every run.

You can also look into the database with DB Browser for SQLite (Open Database → weather.db → Browse Data). Close it before running collect.py again.

Step 7: `collect.py`: the `--loop` option

Some computers cannot use Task Scheduler (e.g. lab PCs that are reset every night). For them we add a second way: the program runs, then sleeps until 08:00 the next day, runs again, and so on.

Change the imports at the top of collect.py to:

import logging
import sys
import time
from datetime import datetime, timedelta

Add this function after run_once():

def seconds_until(hour):
    now = datetime.now()
    next_run = now.replace(hour=hour, minute=0, second=0, microsecond=0)
    if next_run <= now:
        next_run += timedelta(days=1)
    return (next_run - now).total_seconds()

And replace the last part (if __name__ == ...) with:

if __name__ == "__main__":
    run_once()
    if "--loop" in sys.argv:
        while True:
            time.sleep(seconds_until(8))
            run_once()

seconds_until(8) counts how many seconds are left until the next 08:00. If it is already past 08:00 today, it takes tomorrow's 08:00.
sys.argv is the list of words typed after python. python collect.py --loop → ['collect.py', '--loop'].
time.sleep(...) waits that many seconds.

Also update the first lines (the docstring) of the file, so you remember how to use it:

"""Daily job: save tomorrow's forecasts into the database.

    python collect.py          run once (this is what Windows Task Scheduler starts)
    python collect.py --loop   keep running, collect every day at 08:00
"""

Try seconds_until:

python
>>> from datetime import datetime
>>> from collect import seconds_until
>>> datetime.now()
datetime.datetime(2026, 10, 6, 14, 34, 34, 886417)
>>> seconds_until(8) / 3600
17.423642654722222
>>> exit()

At 14:34, the next 08:00 is about 17.4 hours away. Correct.

You do not need to try --loop now (it would wait until tomorrow). If you do, stop it with Ctrl+C.

Step 8: Let Windows run it every morning

We register collect.py in the Task Scheduler with one command. Type it in the terminal (one line; use your paths if your folder is not C:\weather-lab):

schtasks /Create /TN "WeatherLab" /SC DAILY /ST 08:00 /TR "C:\weather-lab\.venv\Scripts\python.exe C:\weather-lab\collect.py"

You should see:

SUCCESS: The scheduled task "WeatherLab" has successfully been created.

What the parts mean: /TN task name, /SC DAILY every day, /ST 08:00 start time, /TR what to run. We use the python.exe inside the venv, so the task has all our packages without activating anything.

Test it now, without waiting for 08:00:

schtasks /Run /TN "WeatherLab"

You should see SUCCESS: Attempted to run the scheduled task "WeatherLab"., and after a few seconds 6 new lines at the end of collect.log with the current time. (No window opens; that is normal.)

Recommended for laptops: open Task Scheduler (Start menu → "Task Scheduler") → Task Scheduler Library → WeatherLab → Properties: - Settings tab → tick "Run task as soon as possible after a scheduled start is missed". Then if the laptop was off at 08:00, the task runs when you switch it on. - Conditions tab → untick "Start the task only if the computer is on AC power".

To remove the task later: schtasks /Delete /TN "WeatherLab" /F

No Task Scheduler? Run python collect.py --loop on a computer that stays on, or use the class database in week 8 (see Downloads).

Your complete files

Compare with yours. If something does not work, copy these.

db.py

"""Everything that touches the SQLite database file (weather.db)."""
import sqlite3
from datetime import datetime

from config import DB_PATH

SCHEMA = """
CREATE TABLE IF NOT EXISTS forecasts (
    city TEXT, source TEXT, day TEXT, tmax REAL, tmin REAL, saved_at TEXT,
    PRIMARY KEY (city, source, day)
);
"""


def connect():
    con = sqlite3.connect(DB_PATH)
    con.executescript(SCHEMA)  # creates the tables the first time, does nothing later
    return con


def now():
    return datetime.now().isoformat(timespec="seconds")


def save_forecasts(city, source, values):
    """values: {day: (tmax, tmin)}. Saving the same day again overwrites it (no duplicates)."""
    rows = [(city, source, day, tmax, tmin, now()) for day, (tmax, tmin) in values.items()]
    con = connect()
    con.executemany("INSERT OR REPLACE INTO forecasts VALUES (?, ?, ?, ?, ?, ?)", rows)
    con.commit()
    con.close()
    return len(rows)

collect.py

"""Daily job: save tomorrow's forecasts into the database.

    python collect.py          run once (this is what Windows Task Scheduler starts)
    python collect.py --loop   keep running, collect every day at 08:00
"""
import logging
import sys
import time
from datetime import datetime, timedelta

from config import BASE_DIR, CITIES
from db import save_forecasts
from sources import PROVIDERS, tomorrow

logging.basicConfig(
    level=logging.INFO, format="%(asctime)s %(message)s",
    handlers=[logging.FileHandler(BASE_DIR / "collect.log", encoding="utf-8"), logging.StreamHandler()],
)
log = logging.getLogger()


def collect_forecasts():
    for key, city in CITIES.items():
        # Live set: one provider failing must not stop the others.
        for source, fetch in PROVIDERS.items():
            try:
                tmax, tmin = fetch(city)
                save_forecasts(key, source, {tomorrow(): (tmax, tmin)})
                log.info(f"{key:9s} {source:11s} {tomorrow()} max {tmax} min {tmin}")
            except Exception as error:
                log.error(f"{key:9s} {source:11s} FAILED: {error}")


def run_once():
    collect_forecasts()


def seconds_until(hour):
    now = datetime.now()
    next_run = now.replace(hour=hour, minute=0, second=0, microsecond=0)
    if next_run <= now:
        next_run += timedelta(days=1)
    return (next_run - now).total_seconds()


if __name__ == "__main__":
    run_once()
    if "--loop" in sys.argv:
        while True:
            time.sleep(seconds_until(8))
            run_once()

sources.py is the same as in week 1.

Check

☐ python collect.py prints 6 lines, no FAILED.
☐ After running it twice, there are still 6 rows (step 6).
☐ schtasks /Run /TN "WeatherLab" adds new lines to collect.log.
☐ Tomorrow: check that the 08:00 run happened: new lines in collect.log, and rows with the next day in the database.

Extra

Write a small script show.py that prints all rows of the forecasts table, ordered by day (ORDER BY day).
Why is saved_at useful? (Hint: what if a service changes its forecast during the day?)

If something goes wrong

Problem	Reason
`database is locked`	DB Browser has the file open with unsaved changes. Close it.
`sqlite3.IntegrityError: UNIQUE constraint failed`	You used `INSERT` instead of `INSERT OR REPLACE`.
Nothing is saved, but no error	`con.commit()` is missing.
The task "ran" but nothing in `collect.log`	Wrong path in `/TR`. Check both paths, and that the `.venv` folder exists.
`ERROR: Access is denied.` from `schtasks`	Your user may not create tasks on this PC (lab PCs). Use `--loop` or your own laptop.
`weather.db` appears in a strange folder	You changed `DB_PATH` to a relative path. Keep `BASE_DIR / "weather.db"`.

Week 2: Saving data and collecting it every day

What you need to know

Step 1: config.py: where the database file is

Step 2: Try SQL by hand

Step 3: db.py: opening the database

Step 4: db.py: saving forecasts

Step 5: collect.py: the daily job

Step 6: Run it again: no duplicates

Step 7: collect.py: the --loop option

Step 8: Let Windows run it every morning

Your complete files

Check

Extra

If something goes wrong

Step 1: `config.py`: where the database file is

Step 3: `db.py`: opening the database

Step 4: `db.py`: saving forecasts

Step 5: `collect.py`: the daily job

Step 7: `collect.py`: the `--loop` option