Compare commits
6 Commits
f3c98b3e46
...
6c0ad94800
| Author | SHA1 | Date | |
|---|---|---|---|
| 6c0ad94800 | |||
| 88ab4756e4 | |||
| 67bdd8b49a | |||
| 149518e65e | |||
| 794723d726 | |||
| e8bce84bf8 |
3
.gitmodules
vendored
3
.gitmodules
vendored
@ -1,3 +1,6 @@
|
||||
[submodule "public"]
|
||||
path = public
|
||||
url = https://gitea.balki.me/balki/blog-op
|
||||
[submodule "themes/mytheme"]
|
||||
path = themes/mytheme
|
||||
url = https://gitea.balki.me/balki/no-js-hugo-theme.git
|
||||
|
||||
15
Makefile
15
Makefile
@ -1,21 +1,10 @@
|
||||
ASCIINEMA_VERSION=3.8.0
|
||||
|
||||
.PHONY: update-bootstrap
|
||||
update-bootstrap:
|
||||
curl -L "https://cdn.jsdelivr.net/npm/bootstrap@5/dist/css/bootstrap.min.css" -o static/assets/bootstrap.min.css
|
||||
curl -L "https://cdn.jsdelivr.net/npm/bootstrap@5/dist/js/bootstrap.min.js" -o static/assets/bootstrap.min.js
|
||||
|
||||
.PHONY: update-asciinema
|
||||
update-asciinema:
|
||||
curl -L "https://github.com/asciinema/asciinema-player/releases/download/v$(ASCIINEMA_VERSION)/asciinema-player.css" -o static/assets/asciinema-player.css
|
||||
curl -L "https://github.com/asciinema/asciinema-player/releases/download/v$(ASCIINEMA_VERSION)/asciinema-player.min.js" -o static/assets/asciinema-player.min.js
|
||||
|
||||
.PHONY: deploy
|
||||
deploy:
|
||||
rsync -av public desk:/var/www/balki.me
|
||||
|
||||
build:
|
||||
rm -rf public/*
|
||||
hugo --logLevel debug
|
||||
|
||||
serve:
|
||||
hugo server -D --bind "0.0.0.0" --navigateToChanged
|
||||
hugo server -D --bind "0.0.0.0" --navigateToChanged -p 2323
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
baseURL: https://balki.me/
|
||||
languageCode: en-us
|
||||
title: Balki's Blog
|
||||
@ -1,9 +1,11 @@
|
||||
---
|
||||
title: "Home"
|
||||
date: 2023-05-31T09:14:56-04:00
|
||||
draft: false
|
||||
---
|
||||
|
||||
Balki's Blog
|
||||
============
|
||||
Just another dev blog.
|
||||
|
||||
* git: https://gitea.balki.me/balki
|
||||
* github: https://github.com/balki
|
||||
* Fediverse: https://social.balki.me/@balki `@balki@balki.me`
|
||||
* RSS: https://blog.balki.me/index.xml
|
||||
|
||||
@ -20,3 +20,16 @@ There should be a asciinema cast above!
|
||||
}
|
||||
```
|
||||
{{< /details >}}
|
||||
|
||||
SWBrqK9fAC6XuZ7eUl3xwnW8SOfekv01TntKdNl/0+noC4Xel+A2elveBYSxh4gWBtMqXR0mT7+8
|
||||
N1jVF/o/Q19m8JF8UC9SiJP01CGDJoROcN2ijdyguLg76A0B2tltU+eqlbm46Pml+Ukndda1wxdY
|
||||
eWs1favc/qsN2txx5jL/6srrEEVIKr3p6ov4hOVtju0dswaseknxZagMfaBwbk6KNGS9r5Nmk+MY
|
||||
0jdYOfcBIHGB/IxvifV+ux5l/C7VK/PBFOm6t/qtVbczDiPtzngOcsR/CQOej5Qfi+2mXEVbci53
|
||||
GMois/cjJF4vSU7/OcdxCkkLpN42UKyFOPiR3TczXrsM0kt1Zl/y8huf9KLkRC61MD2Zn/FOvl4U
|
||||
Jsd9CsVNqN2DO5ABxHqSJ5oI9d03rJilWFunJ6jURH8CyfQv5vJ8ORQmMZdBF1DaeXm6K7/R26lY
|
||||
ArRpYtgWNhzDWhyGSM2E3/XGvE1w4iZ/u8ml8K6R+tInxYRd1+6ScdTr0NgweuXbnk/27G66+NVv
|
||||
LX3t9d+rWvwdJBzibimLzy6Rnxq6XO1CSmBVTagr4edyJe76GqSS2RMhk3zbdUyP5gyPrYyCDQn5
|
||||
/0rQsHmzGd9izC6Gjuk7hu10Y3eWi5DCfrDGfyrADO90uYqXrW2tjliwdRJuZtSuRxbOVaY3G3UX
|
||||
MPG6pRH/byyKTduN2RS4VNo73ZkQOOooQ3NQErGPc29yeWdvLoA7DB5rsMf7V0XZcpLrUVM6SoQf
|
||||
|
||||
<!-- vim: set lw=DETAILS_FDM: -->
|
||||
|
||||
209
content/posts/gitea-mirror.md
Normal file
209
content/posts/gitea-mirror.md
Normal file
@ -0,0 +1,209 @@
|
||||
---
|
||||
title: "Mirroring git repositories with Gitea"
|
||||
date: 2024-06-27T10:05:46-04:00
|
||||
draft: true
|
||||
tags:
|
||||
- gitea
|
||||
- curl
|
||||
- yq
|
||||
categories:
|
||||
- development
|
||||
---
|
||||
|
||||
[Gitea][1] is an awesome self hosted git forge. I use the [pull-mirror][2]
|
||||
feature to mirror many git repos (mostly from github). In this post, I want to
|
||||
share few maintenance scripts I ran that connects to [gitea-api][14] using
|
||||
[yq][] and [curl][].
|
||||
|
||||
<!--more-->
|
||||
|
||||
# TODO
|
||||
|
||||
* Change mirror interval
|
||||
* Disable for mirror repos by default actions
|
||||
* use jo https://github.com/jpmens/jo
|
||||
* get just latest mirror and apply the settings
|
||||
|
||||
## Setup
|
||||
|
||||
### API token
|
||||
|
||||
Go to `https://gitea.balki.me/user/settings/applications` and Generate new
|
||||
Token with write permission for repositories.
|
||||
|
||||
# TODO
|
||||
set base url
|
||||
|
||||
## Why Mirror?
|
||||
|
||||
1. Remote sites may disappear one day.
|
||||
2. Better local code search. Github does not allow code search without signing
|
||||
in.
|
||||
3. Tools like vim-plugins can use the local urls which is better for privacy.
|
||||
4. Setup notification when a repository creates a new tag.
|
||||
|
||||
## Dumb crawlers problem
|
||||
|
||||
My gitea [instance][3] is public and had all mirror repos public as well. This
|
||||
caused a huge network traffic from bots.
|
||||
|
||||
I created an [organization][4] without public visibility and made it own all
|
||||
the mirror repos.
|
||||
|
||||
```
|
||||
❯ yq --version
|
||||
yq (https://github.com/mikefarah/yq/) version v4.44.1
|
||||
|
||||
❯ curl -V | head -c 11
|
||||
curl 8.8.0
|
||||
```
|
||||
|
||||
## API token
|
||||
|
||||
## Hooks for notification
|
||||
[Hooks][5]
|
||||
|
||||
Lets first download the list of all mirror repos
|
||||
|
||||
```bash
|
||||
TOKEN=d88446542e844f4da4ba75bbb85bd694a71907b5
|
||||
|
||||
curl "https://gitea.balki.me/api/v1/repos/search?limit=100&mode=mirror" \
|
||||
-H "accept: application/json" \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-o mirror-repos.json
|
||||
```
|
||||
|
||||
References
|
||||
|
||||
1. API doc: https://gitea.balki.me/api/swagger#/repository/repoSearch
|
||||
|
||||
Create a [hook][5] manually in one repo and get the hook using the API
|
||||
|
||||
```bash
|
||||
curl -s "https://gitea.balki.me/api/v1/repos/MirrorWatch/snac2/hooks?page=1&limit=10" \
|
||||
-H "Authorization: token $TOKEN" | yq -P -oj
|
||||
```
|
||||
|
||||
Sample Output:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 32,
|
||||
"type": "telegram",
|
||||
"branch_filter": "tag",
|
||||
"config": {
|
||||
"content_type": "json",
|
||||
"url": "https://api.telegram.org/bot1169894068:J1JVbV3f2vEQpdnPqFANfhjWZrFuUCJs1EW/sendMessage?chat_id=-1008910751069"
|
||||
},
|
||||
"events": [
|
||||
"create"
|
||||
],
|
||||
"authorization_header": "",
|
||||
"active": true,
|
||||
"updated_at": "2024-06-20T20:54:33-04:00",
|
||||
"created_at": "2024-06-20T20:54:33-04:00"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Now loop through all mirror repos and add the same webook. Remove unwanted fields like `id`, `created_at`, etc.,
|
||||
|
||||
```bash
|
||||
yq -r '.data[] | .full_name' mirror-repos.json | while read -r repo; do
|
||||
echo "$repo"
|
||||
|
||||
curl "https://gitea.balki.me/api/v1/repos/$repo/hooks" \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
--json @- <<-EOM
|
||||
{
|
||||
"active": true,
|
||||
"branch_filter": "tag",
|
||||
"config": {
|
||||
"content_type": "json",
|
||||
"url": "https://api.telegram.org/bot1169894068:J1JVbV3f2vEQpdnPqFANfhjWZrFuUCJs1EW/sendMessage?chat_id=-1008910751069"
|
||||
},
|
||||
"events": [
|
||||
"create"
|
||||
],
|
||||
"type": "telegram"
|
||||
}
|
||||
EOM
|
||||
|
||||
echo "============"
|
||||
done
|
||||
```
|
||||
|
||||
## Fixing issues and pr links
|
||||
|
||||
Fix the issue url setting in first repo as shown [here][7].
|
||||
|
||||
Get the json representataion.
|
||||
|
||||
```bash
|
||||
yq '.data[] | .external_tracker ' mirror-repos.json | head
|
||||
```
|
||||
|
||||
Sample output
|
||||
|
||||
```json
|
||||
{
|
||||
"external_tracker_url": "https://github.com/caddyserver/caddy/issues",
|
||||
"external_tracker_format": "https://github.com/caddyserver/caddy/issues/{index}",
|
||||
"external_tracker_style": "numeric",
|
||||
"external_tracker_regexp_pattern": ""
|
||||
}
|
||||
```
|
||||
|
||||
Now loop throug all repos and update. Making sure only add to github repos and
|
||||
they are not already updated
|
||||
|
||||
```bash
|
||||
yq -r '.data[]
|
||||
| select(.original_url == "*github*" and has("internal_tracker") )
|
||||
| "\(.full_name) \(.original_url)"' mirror-repos.json | while read -r repo og; do
|
||||
echo "Repo is $repo and github origin url is $og"
|
||||
curl "https://gitea.balki.me/api/v1/repos/$repo" \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-X PATCH \
|
||||
--json @- <<-EOM
|
||||
{
|
||||
"external_tracker": {
|
||||
"external_tracker_url": "${og%.git}/issues",
|
||||
"external_tracker_format": "${og%.git}/issues/{index}",
|
||||
"external_tracker_style": "numeric",
|
||||
"external_tracker_regexp_pattern": ""
|
||||
}
|
||||
}
|
||||
EOM
|
||||
done
|
||||
yq -r '.data[] | .full_name' mirror-repos.json | while read -r repo; do
|
||||
echo "$repo"
|
||||
jo has_actions=false | curl "https://gitea.balki.me/api/v1/repos/$repo" \
|
||||
-H "Authorization: token $TOKEN" \
|
||||
-X PATCH \
|
||||
--json @-
|
||||
done
|
||||
```
|
||||
|
||||
### Doc links
|
||||
* yq: [select][8], [has][9], [string interpolation][10]
|
||||
* bash: [parameter expansion][11], [here-doc][12]
|
||||
* curl: [`--json`][13]
|
||||
|
||||
|
||||
[1]: https://github.com/go-gitea/gitea
|
||||
[2]: https://docs.gitea.com/usage/repo-mirror#pulling-from-a-remote-repository
|
||||
[3]: https://gitea.balki.me
|
||||
[4]: https://docs.gitea.com/usage/permissions#organization-repository
|
||||
[5]: https://docs.gitea.com/usage/webhooks
|
||||
[6]: https://docs.gitea.com/development/api-usage
|
||||
[7]: https://github.com/go-gitea/gitea/issues/18986
|
||||
[8]: https://mikefarah.gitbook.io/yq/operators/select
|
||||
[9]: https://mikefarah.gitbook.io/yq/operators/has#select-checking-for-existence-of-deep-paths
|
||||
[10]: https://mikefarah.gitbook.io/yq/operators/string-operators#interpolation
|
||||
[11]: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
|
||||
[12]: https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Here-Documents
|
||||
[13]: https://everything.curl.dev/http/post/json.html
|
||||
[14]: https://gitea.balki.me/api/swagger#/repository/repoSearch
|
||||
|
||||
@ -101,7 +101,7 @@ autocmd BufNewFile,BufRead cheat.sh nnoremap <buffer> <silent><2-LeftMouse> :Sen
|
||||
|
||||
### Demo
|
||||
|
||||
{{< asciinema key="vimstt" >}}
|
||||
{{< asciinema key="vimstt" poster="npt:0:23" startAt="0:18" >}}
|
||||
|
||||
### Using
|
||||
|
||||
13
hugo.yaml
Normal file
13
hugo.yaml
Normal file
@ -0,0 +1,13 @@
|
||||
baseURL: https://blog.balki.me/
|
||||
languageCode: en-us
|
||||
title: Balki's Blog
|
||||
theme: mytheme
|
||||
markup:
|
||||
highlight:
|
||||
# This setting makes code highlighting use theme's colors instead of hugo's default
|
||||
noClasses: false
|
||||
|
||||
params:
|
||||
themeStyleSwitcher: true
|
||||
footerText: "[theme](https://gitea.balki.me/balki/no-js-hugo-theme)"
|
||||
showTagList: false
|
||||
@ -1,58 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta http-equiv="Content-Security-Policy" content="default-src 'self' 'unsafe-inline' data: ;{{ if .Params.asciinema }}script-src 'self' 'unsafe-inline' 'unsafe-eval';{{ end }}">
|
||||
<link rel="alternate" type="application/rss+xml" href="/posts/index.xml">
|
||||
<title>
|
||||
{{- block "title" . -}}
|
||||
{{ .Site.Title }}
|
||||
{{- end -}}
|
||||
</title>
|
||||
|
||||
<!-- Diable favicon requests: https://stackoverflow.com/a/13416784 -->
|
||||
<link rel="icon" href="data:;base64,iVBORw0KGgo=">
|
||||
<link href="/assets/bootstrap.min.css" rel="stylesheet" >
|
||||
{{ if .Params.asciinema }}
|
||||
<link rel="stylesheet" type="text/css" href="/assets/asciinema-player.css" />
|
||||
{{ end }}
|
||||
<style>
|
||||
{{ block "styles" . }}
|
||||
{{ end }}
|
||||
div.highlight pre {
|
||||
{{/* code highlighting */}}
|
||||
padding: 1rem;
|
||||
}
|
||||
</style>
|
||||
|
||||
<script src="/assets/bootstrap.min.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
<div class="container" >
|
||||
<aside>
|
||||
<nav>
|
||||
<ul class="nav">
|
||||
<li class="nav-item"><a class="nav-link" href="/"> Home </a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/posts"> Blog </a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="https://gitea.balki.me/balki"> Gitea </a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="https://github.com/balki"> Github </a></li>
|
||||
<!-- TODO
|
||||
<li class="nav-item"><a class="nav-link" href="/about"> About </a></li>
|
||||
-->
|
||||
</ul>
|
||||
</nav>
|
||||
</aside>
|
||||
|
||||
<main>
|
||||
{{ block "main" . }}
|
||||
{{ end }}
|
||||
</main>
|
||||
<hr>
|
||||
<div style="height: 300px;"><!-- This space is intentionally left blank --></div>
|
||||
</div>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@ -1,6 +0,0 @@
|
||||
{{ define "main" }}
|
||||
<h1 class="text-center">Posts</h1>
|
||||
{{ range .Pages }}
|
||||
{{ partial "postcard" . }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
@ -1,24 +0,0 @@
|
||||
{{ define "title" }}
|
||||
{{ .Title }} – {{ .Site.Title }}
|
||||
{{ end }}
|
||||
{{ define "main" }}
|
||||
<article>
|
||||
<h1 class="display-3 text-center">{{ .Title }}</h1>
|
||||
<p class="text-center"><time datetime="{{ .Date }}">{{ .Date.Format "2006-01-02" }}</time>
|
||||
{{ if .Params.tags }}
|
||||
│
|
||||
{{range .Params.tags}}
|
||||
<a class="btn btn-primary" href="/tags/{{ . | urlize }}" role="button">{{ . }}</a>
|
||||
{{end}}
|
||||
{{end}}
|
||||
{{ if .Params.categories }}
|
||||
│
|
||||
{{range .Params.categories}}
|
||||
<a class="btn btn-secondary" href="/categories/{{ . | urlize }}" role="button">{{ . }}</a>
|
||||
{{end}}
|
||||
{{end}}
|
||||
</p>
|
||||
{{ .Content }}
|
||||
</article>
|
||||
{{ end }}
|
||||
|
||||
@ -1,16 +0,0 @@
|
||||
{{ define "main" }}
|
||||
<article>
|
||||
<div class="text-center m-4">
|
||||
{{ .Content }}
|
||||
</div>
|
||||
<p class="text-center">Under construction. Come back later! :)</p>
|
||||
<div>
|
||||
<h4>Recent posts</h4>
|
||||
{{ range .Pages }}
|
||||
{{ range .Pages }}
|
||||
{{ partial "postcard" . }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</div>
|
||||
</article>
|
||||
{{ end }}
|
||||
@ -1,12 +0,0 @@
|
||||
{{ define "styles" }}
|
||||
a.title-link {
|
||||
text-decoration: none;
|
||||
}
|
||||
{{ end }}
|
||||
<div class="card mb-4">
|
||||
<div class="card-body">
|
||||
<a class="title-link link-primary" href="{{ .RelPermalink }}" ><h3 class="card-title">{{ .Title }}</h3></a>
|
||||
<p> {{ .Date.Format "2006-01-02" }} </p>
|
||||
<p class="card-text"> {{ .Summary }} </p>
|
||||
</div>
|
||||
</div>
|
||||
@ -1,20 +0,0 @@
|
||||
<div id='{{ with .Get "key" }}{{ . }}{{ end }}'></div>
|
||||
<script src="/assets/asciinema-player.min.js"></script>
|
||||
<script>
|
||||
AsciinemaPlayer.create('/casts/{{ with .Get "key" }}{{ . }}{{ end }}.cast', document.getElementById('{{ with .Get "key" }}{{ . }}{{ end }}'), {
|
||||
{{ if .Get "cols" }}cols: "{{ .Get "cols" }}",{{ end }}
|
||||
{{ if .Get "rows" }}rows: "{{ .Get "rows" }}",{{ end }}
|
||||
{{ if .Get "autoPlay" }}autoPlay: "{{ .Get "autoPlay" }}",{{ end }}
|
||||
{{ if .Get "preload" }}preload: "{{ .Get "preload" }}",{{ end }}
|
||||
{{ if .Get "loop" }}loop: "{{ .Get "loop" }}",{{ end }}
|
||||
{{ if .Get "startAt" }}startAt: "{{ .Get "startAt" }}",{{ end }}
|
||||
{{ if .Get "speed" }}speed: "{{ .Get "speed" }}",{{ end }}
|
||||
{{ if .Get "idleTimeLimit" }}idleTimeLimit: "{{ .Get "idleTimeLimit" }}",{{ end }}
|
||||
{{ if .Get "theme" }}theme: "{{ .Get "theme" }}",{{ end }}
|
||||
{{ if .Get "poster" }}poster: "{{ .Get "poster" }}",{{ end }}
|
||||
{{ if .Get "fit" }}fit: "{{ .Get "fit" }}",{{ end }}
|
||||
{{ if .Get "terminalFontSize" }}terminalFontSize: "{{ .Get "terminalFontSize" }}",{{ end }}
|
||||
{{ if .Get "terminalFontFamily" }}terminalFontFamily: "{{ .Get "terminalFontFamily" }}",{{ end }}
|
||||
{{ if .Get "terminalLineHeight" }}terminalLineHeight: "{{ .Get "terminalLineHeight" }}",{{ end }}
|
||||
});
|
||||
</script>
|
||||
@ -1,9 +0,0 @@
|
||||
{{- /*
|
||||
Template comment syntax: https://hugo.d.balki.me/templates/introduction/#comments
|
||||
collapsable section ref: https://stackoverflow.com/a/78475534
|
||||
*/ -}}
|
||||
|
||||
<details>
|
||||
<summary>{{ .Get "title" | default "Click to expand" | markdownify }}</summary>
|
||||
<div>{{ .Inner | markdownify }}</div>
|
||||
</details>
|
||||
2
public
2
public
Submodule public updated: 094b566f7e...15935f461a
1
themes/mytheme
Submodule
1
themes/mytheme
Submodule
Submodule themes/mytheme added at c93379bf7c
Reference in New Issue
Block a user