fix email links, add deploy script, update README
This commit is contained in:
flo-eberle
274
README.md
274
README.md
@@ -35,6 +35,7 @@ flat_scraper/
|
||||
├── .gitignore # Git ignore für .env und data/
|
||||
├── requirements.txt # Python Dependencies
|
||||
├── Dockerfile # ARM64 optimiert
|
||||
├── deploy.sh # Automatisches Deployment Script
|
||||
├── docker-compose.yml # Automatisierung
|
||||
└── README.md
|
||||
```
|
||||
@@ -174,6 +175,232 @@ username: "${EMAIL_USERNAME:default@example.com}" # Mit Default
|
||||
|
||||
## Deployment auf Raspberry Pi
|
||||
|
||||
### Schritt-für-Schritt Anleitung
|
||||
|
||||
#### Voraussetzungen
|
||||
- Raspberry Pi 4+ (empfohlen)
|
||||
- Docker & Docker Compose installiert
|
||||
- Git für das Klonen des Repositories
|
||||
|
||||
#### 1. Repository auf Raspberry Pi laden
|
||||
|
||||
```bash
|
||||
# Projekt klonen
|
||||
git clone <dein-repo-url> flat_scraper
|
||||
cd flat_scraper
|
||||
|
||||
# Oder manuell kopieren
|
||||
scp -r ./flat_scraper pi@raspberry-pi:~/
|
||||
```
|
||||
|
||||
#### 2. Environment konfigurieren
|
||||
|
||||
```bash
|
||||
# Environment Vorlage kopieren
|
||||
cp .env.example .env
|
||||
|
||||
# Mit Editor öffnen
|
||||
nano .env
|
||||
# Oder vim .env
|
||||
```
|
||||
|
||||
**Deine SMTP-Daten eintragen:**
|
||||
```bash
|
||||
SMTP_SERVER=dein-smtp-server.com
|
||||
SMTP_PORT=587
|
||||
EMAIL_USERNAME=deine-email@domain.com
|
||||
EMAIL_PASSWORD=dein-app-password
|
||||
EMAIL_FROM=deine-email@domain.com
|
||||
EMAIL_TO=empfänger@domain.com
|
||||
EMAIL_SECURITY=starttls
|
||||
```
|
||||
|
||||
#### 3. Docker Image bauen
|
||||
|
||||
```bash
|
||||
# Image für ARM64 bauen (kann einige Minuten dauern)
|
||||
docker build -t flat-scraper .
|
||||
|
||||
# Build-Status prüfen
|
||||
docker images | grep flat-scraper
|
||||
```
|
||||
|
||||
**Fehlerbehandlung:**
|
||||
```bash
|
||||
# Falls Docker Daemon nicht läuft:
|
||||
sudo systemctl start docker
|
||||
sudo systemctl enable docker
|
||||
|
||||
# Falls Podman-Socket Fehler:
|
||||
# Prüfen ob Docker oder Podman aktiv ist:
|
||||
which docker
|
||||
which podman
|
||||
|
||||
# Docker Socket prüfen:
|
||||
ls -la /var/run/docker.sock
|
||||
sudo chmod 666 /var/run/docker.sock
|
||||
|
||||
# Falls nötig, Docker neu starten:
|
||||
sudo systemctl restart docker
|
||||
|
||||
# Environment Variablen Fehler:
|
||||
# Prüfen ob .env Datei existiert und korrekt ist:
|
||||
ls -la .env
|
||||
cat .env
|
||||
|
||||
# Falls SMTP_PORT nicht gesetzt ist:
|
||||
echo "SMTP_PORT=587" >> .env
|
||||
|
||||
# Permission Fehler mit data/ Verzeichnis:
|
||||
sudo chown -R $USER:$USER data/
|
||||
chmod 755 data/
|
||||
```
|
||||
|
||||
#### 4. Erster Testlauf
|
||||
|
||||
```bash
|
||||
# Einmaligen Scraper-Lauf testen
|
||||
docker run --rm \
|
||||
--env-file $(pwd)/.env \
|
||||
-v $(pwd)/data:/app/data \
|
||||
-v $(pwd)/src/config:/app/src/config \
|
||||
flat-scraper python src/main.py
|
||||
```
|
||||
|
||||
**Erwartete Ausgabe:**
|
||||
```
|
||||
2026-02-15 10:30:00 - INFO - Starting flat scraper run
|
||||
2026-02-15 10:30:00 - INFO - Start scraping nhg
|
||||
Scraping PLZ 1120 Wien...
|
||||
Scraping PLZ 1140 Wien...
|
||||
Scraping PLZ 1150 Wien...
|
||||
Scraping PLZ 1160 Wien...
|
||||
2026-02-15 10:30:15 - INFO - Found X results for nhg
|
||||
Email-Benachrichtigung gesendet an 1 Empfänger
|
||||
2026-02-15 10:30:15 - INFO - Scraping completed: 1/1 sites successful
|
||||
```
|
||||
|
||||
#### 5. Docker Compose konfigurieren
|
||||
|
||||
```bash
|
||||
# ARM64 Support aktivieren (wichtig für Raspberry Pi)
|
||||
sed -i 's/# platform: linux/arm64/platform: linux/arm64/' docker-compose.yml
|
||||
|
||||
# Konfiguration prüfen
|
||||
cat docker-compose.yml
|
||||
```
|
||||
|
||||
#### 6. Production Deployment
|
||||
|
||||
```bash
|
||||
# Manuelles Deployment
|
||||
docker-compose up -d
|
||||
|
||||
# Status prüfen
|
||||
docker-compose ps
|
||||
|
||||
# ODER: Automatisches Deployment Script verwenden
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
**Deployment Script Features:**
|
||||
- ✅ **Automatisches Build** mit Fehlerprüfung
|
||||
- ✅ **Image Cleanup** für Speicherplatz
|
||||
- ✅ **Status Überwachung** vor/nach Deployment
|
||||
- ✅ **Flexible Optionen** für verschiedene Szenarien
|
||||
|
||||
```bash
|
||||
# Vollständiges Deployment mit Cleanup
|
||||
./deploy.sh
|
||||
|
||||
# Deployment ohne Cleanup (für schnelle Tests)
|
||||
./deploy.sh --no-cleanup
|
||||
|
||||
# Nur Services neu starten (kein Build)
|
||||
./deploy.sh --no-build
|
||||
|
||||
# Verbose Output für Debugging
|
||||
./deploy.sh --verbose
|
||||
|
||||
# Hilfe anzeigen
|
||||
./deploy.sh --help
|
||||
```
|
||||
|
||||
**Erwartete Ausgabe:**
|
||||
```
|
||||
NAME COMMAND SERVICE STATUS PORTS
|
||||
flat_scraper-flat-scraper-1 "python src/main.py" flat-scraper running
|
||||
flat_scraper-scheduler-1 "python -c 'import..." scheduler running
|
||||
```
|
||||
|
||||
#### 7. Logs überwachen
|
||||
|
||||
```bash
|
||||
# Live Logs ansehen
|
||||
docker-compose logs -f
|
||||
|
||||
# Nur Scraper Logs
|
||||
docker-compose logs -f flat-scraper
|
||||
|
||||
# Nur Scheduler Logs
|
||||
docker-compose logs -f scheduler
|
||||
```
|
||||
|
||||
#### 8. Automatisierung verifizieren
|
||||
|
||||
```bash
|
||||
# Prüfen ob Scheduler läuft
|
||||
docker-compose logs scheduler | grep "Scheduler started"
|
||||
|
||||
# Nächsten Lauf prüfen (alle 6 Stunden)
|
||||
docker-compose logs scheduler | tail -10
|
||||
```
|
||||
|
||||
#### 9. Daten persistenz prüfen
|
||||
|
||||
```bash
|
||||
# CSV Dateien prüfen
|
||||
ls -la data/
|
||||
|
||||
# Inhalt ansehen
|
||||
cat data/nhg_results.csv
|
||||
|
||||
# Letzte Ergebnisse
|
||||
tail -5 data/nhg_results.csv
|
||||
```
|
||||
|
||||
#### 10. Wartung und Updates
|
||||
|
||||
```bash
|
||||
# Services stoppen
|
||||
docker-compose down
|
||||
|
||||
# Code aktualisieren
|
||||
git pull origin main
|
||||
|
||||
# Automatisches Deployment mit Script
|
||||
./deploy.sh
|
||||
|
||||
# ODER manueller Weg:
|
||||
# Neues Image bauen
|
||||
docker build -t flat-scraper .
|
||||
|
||||
# Services neu starten
|
||||
docker-compose up -d
|
||||
|
||||
# Alte Images aufräumen
|
||||
docker image prune -f
|
||||
```
|
||||
|
||||
**Deployment Script für Updates:**
|
||||
```bash
|
||||
# Vollständiges Update mit Cleanup
|
||||
./deploy.sh
|
||||
|
||||
# Schneller Restart ohne Build
|
||||
./deploy.sh --no-build --no-cleanup
|
||||
```
|
||||
|
||||
### ARM64 Support
|
||||
|
||||
Der Dockerfile ist für ARM64 optimiert:
|
||||
@@ -206,6 +433,53 @@ RUN apt-get update && apt-get install -y chromium
|
||||
2. **Keine Ergebnisse**: PLZ nicht verfügbar oder Website geändert
|
||||
3. **Email funktioniert nicht**: SMTP-Einstellungen und Security prüfen
|
||||
4. **Environment nicht geladen**: `.env` Datei prüfen und Rechte
|
||||
5. **Docker Socket Fehler**: Podman vs Docker Konflikt
|
||||
6. **TypeError: int() argument must be a string**: Environment Variable fehlt oder hat Default-Wert nicht
|
||||
7. **Permission denied: data/nhg_results.csv**: data/ Verzeichnis gehört falschem User
|
||||
|
||||
### Docker/Podman Konflikt lösen
|
||||
|
||||
```bash
|
||||
# 1. Prüfen welcher Container-Manager aktiv:
|
||||
echo $CONTAINER_MANAGER
|
||||
which docker
|
||||
which podman
|
||||
|
||||
# 2. Docker als Standard setzen (falls nötig):
|
||||
export DOCKER_HOST=unix:///var/run/docker.sock
|
||||
|
||||
# 3. Podman deaktivieren (falls gewünscht):
|
||||
sudo systemctl disable podman
|
||||
sudo systemctl stop podman
|
||||
|
||||
# 4. Docker Socket Rechte prüfen:
|
||||
sudo ls -la /var/run/docker.sock
|
||||
sudo usermod -aG docker $USER # User zur Docker Gruppe hinzufügen
|
||||
newgrp docker # Gruppe neu laden
|
||||
|
||||
# 5. System neu starten nach Änderungen:
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
### Raspberry Pi spezifische Probleme
|
||||
|
||||
```bash
|
||||
# ARM64 Architektur prüfen:
|
||||
uname -m
|
||||
# Sollte: aarch64 oder armv7l
|
||||
|
||||
# Docker Architektur prüfen:
|
||||
docker version --format '{{.Server.Arch}}'
|
||||
|
||||
# Falls x86_64 Images auf ARM64 laufen sollen:
|
||||
# docker run --platform linux/arm64 flat-scraper
|
||||
|
||||
# Memory prüfen (mindestens 1GB empfohlen):
|
||||
free -h
|
||||
|
||||
# Speicherplatz prüfen (mindestens 2GB frei):
|
||||
df -h
|
||||
```
|
||||
|
||||
### Debugging
|
||||
|
||||
|
||||
Reference in New Issue
Block a user