Read-only DB access for the Data Lake gold pull

01 Connect

JDBC URL

jdbc:postgresql://sps-v4-prod-db.postgres.database.azure.com:5432/sps_v4?sslmode=require

Databricks (the Postgres driver is bundled — no extra install)

url = "jdbc:postgresql://sps-v4-prod-db.postgres.database.azure.com:5432/sps_v4?sslmode=require"
props = {
    "user": "pbi_reader",
    "password": dbutils.secrets.get(scope="...", key="pbi_reader_pw"),
    "driver": "org.postgresql.Driver",
}
df = spark.read.jdbc(url=url, table="pbi_v1.v_projects", properties=props)

02 The contract

• Read only pbi_v1.*. These views are a stable façade. Internal column renames on the app's real tables are absorbed here — they will not break your pipeline.
• Columns are add-only. A view may gain columns without notice. A column is never removed or renamed inside pbi_v1 — that would ship as a new pbi_v2 schema so your existing reads keep working.
• v1 is full-refresh. Only v_projects.updated_at and v_project_status_weekly.submitted_at are real row-level timestamps. Everything else exposes only its natural grain (month, year/week_number) as a coarse watermark. Do not assume row-level change deltas. If you need true incremental loading, that's a pbi_v2 request (see §07).
• Sensitive data is not exposed. Project/financial/status base tables, users, auth, audit, drafts and touch logs are all invisible to pbi_reader by design.

03 Column dictionary

Seven views. project_id is text and joins across all of them. Types are the Postgres types you'll receive.

pbi_v1.v_projects

One row per non-archived project. Master fields as seen on the STATUSAS / PROJEKTAI tabs. grain: project_id

pbi_v1.v_project_status_weekly

One row per project per ISO week — the full weekly STATUSAS history. grain: project_id, year, week_number

pbi_v1.v_latest_project_status_weekly

The single most-recent weekly row per project. Same columns as above, plus one. grain: project_id

pbi_v1.v_project_financials

One row per project per month. grain: project_id, month

pbi_v1.v_project_materials_monthly

One row per project per material per month. grain: project_id, material_name, month

pbi_v1.v_project_work_volumes

One row per project per ISO week. grain: project_id, year, week_number

pbi_v1.v_project_critical_activities

One row per active critical activity per project (current state, archived rows excluded). grain: project_id, sequence

04 Sample query

Latest weekly status across all active projects — mirrors the STATUSAS browse screen.

SELECT p.code, p.name, p.stage, p.country, p.pv_name,
       sw.expected_end_date, sw.workers_hotrema, sw.workers_other,
       sw.sukaupimai, sw.atsargos_viso, sw.atsargos_30d, sw.atsargos_90d,
       sw.year, sw.week_number, sw.is_current_week
FROM pbi_v1.v_projects p
JOIN pbi_v1.v_latest_project_status_weekly sw ON sw.project_id = p.project_id
ORDER BY p.code;

05 Things that bite

• ISO weeks, not calendar weeks. year/week_number and iso_week_key are ISO — at year boundaries an early-January date can belong to week 52/53 of the previous ISO year. Join on iso_week_key or on the (year, week_number) pair, never on a calendar-derived week.
• Financial month is month-end. The date is the last day of the month, by design, so roll-ups stay canonical. Don't assume first-of-month.
• Percent columns may exceed 100. buffer_consumed_pct routinely does. Treat them as raw percent values, not clamped ratios.
• Most metric columns are nullable. A missing week/field is null, not 0 — coalesce deliberately where a 0 is meant.
• Material categories are not fixed. material_type is admin-managed and grows over time; key measures off unit. (The old VATA / SKARDA / KEVALAI trio is gone.)
• Names can be comma-joined. A project may have several PV/AV/PS people, so pv_name can read "Jonas J., Petras P.". Split if you need them individually.
• Units marked ⚑ are unconfirmed. Assumed EUR / m² from precision and naming — confirm with Vilius before wiring them into money or area measures.

06 Load & refresh

• Full-refresh is fine today. ~140 projects — the whole set is small. Scheduling the job is on your side.
• Run off-hours, keep partitions modest. Prod is a Burstable B2s shared with the live app. A nightly full pull won't disturb it; a wide concurrent read during business hours could.
• Also readable — non-PII code-lists via existing public grants, handy as gold labels (not part of pbi_v1, so no stability promise, but stable in practice): reference.project_stages, reference.deviation_reasons, reference.eu_holidays, public.material_types, public.materials_catalog.

07 Changes & contact

• Need a new column? Ask Vilius — columns are added to a pbi_v1 view without breaking existing reads.
• Need row-level incremental loading (real change deltas)? That's a pbi_v2 with added timestamp columns — raise it and it gets scoped separately.
• Connection times out? Almost always the network prerequisite in §01: an executor node is egressing from an IP that isn't allow-listed, or the cluster's egress moved. A NAT gateway on the workspace keeps it to one stable IP; otherwise send Vilius the current egress IP to refresh. Auth failures (wrong password) are separate — check the secret.