update existing

2025-05-20 23:13:03 -04:00
parent f3c98b3e46
commit e8bce84bf8
5 changed files with 224 additions and 3 deletions
--- a/2
+++ b/2
@ -18,4 +18,4 @@ build:
 	hugo --logLevel debug

 serve:
-	hugo server -D --bind "0.0.0.0" --navigateToChanged
+	hugo server -D --bind "0.0.0.0" --navigateToChanged -p 2323
--- a/content/pages/playground.md
+++ b/content/pages/playground.md
@ -20,3 +20,16 @@ There should be a asciinema cast above!
 }
 ```
 {{< /details >}}
+
+SWBrqK9fAC6XuZ7eUl3xwnW8SOfekv01TntKdNl/0+noC4Xel+A2elveBYSxh4gWBtMqXR0mT7+8
+N1jVF/o/Q19m8JF8UC9SiJP01CGDJoROcN2ijdyguLg76A0B2tltU+eqlbm46Pml+Ukndda1wxdY
+eWs1favc/qsN2txx5jL/6srrEEVIKr3p6ov4hOVtju0dswaseknxZagMfaBwbk6KNGS9r5Nmk+MY
+0jdYOfcBIHGB/IxvifV+ux5l/C7VK/PBFOm6t/qtVbczDiPtzngOcsR/CQOej5Qfi+2mXEVbci53
+GMois/cjJF4vSU7/OcdxCkkLpN42UKyFOPiR3TczXrsM0kt1Zl/y8huf9KLkRC61MD2Zn/FOvl4U
+Jsd9CsVNqN2DO5ABxHqSJ5oI9d03rJilWFunJ6jURH8CyfQv5vJ8ORQmMZdBF1DaeXm6K7/R26lY
+ArRpYtgWNhzDWhyGSM2E3/XGvE1w4iZ/u8ml8K6R+tInxYRd1+6ScdTr0NgweuXbnk/27G66+NVv
+LX3t9d+rWvwdJBzibimLzy6Rnxq6XO1CSmBVTagr4edyJe76GqSS2RMhk3zbdUyP5gyPrYyCDQn5
+/0rQsHmzGd9izC6Gjuk7hu10Y3eWi5DCfrDGfyrADO90uYqXrW2tjliwdRJuZtSuRxbOVaY3G3UX
+MPG6pRH/byyKTduN2RS4VNo73ZkQOOooQ3NQErGPc29yeWdvLoA7DB5rsMf7V0XZcpLrUVM6SoQf
+
+<!-- vim: set lw=DETAILS_FDM: -->
--- a/content/posts/gitea-mirror.md
+++ b/content/posts/gitea-mirror.md
@ -0,0 +1,209 @@
+---
+title: "Mirroring git repositories with Gitea"
+date: 2024-06-27T10:05:46-04:00
+draft: true
+tags:
+    - gitea
+    - curl
+    - yq
+categories:
+    - development
+---
+
+[Gitea][1] is an awesome self hosted git forge. I use the [pull-mirror][2]
+feature to mirror many git repos (mostly from github). In this post, I want to
+share few maintenance scripts I ran that connects to [gitea-api][14] using
+[yq][] and [curl][].
+
+<!--more-->
+
+# TODO
+
+  * Change mirror interval
+  * Disable for mirror repos by default actions
+  * use jo https://github.com/jpmens/jo
+  * get just latest mirror and apply the settings
+
+## Setup
+
+### API token
+
+Go to `https://gitea.balki.me/user/settings/applications` and Generate new
+Token with write permission for repositories.
+
+# TODO
+set base url
+
+## Why Mirror?
+
+1. Remote sites may disappear one day.
+2. Better local code search. Github does not allow code search without signing
+   in.
+3. Tools like vim-plugins can use the local urls which is better for privacy.
+4. Setup notification when a repository creates a new tag.
+
+## Dumb crawlers problem
+
+My gitea [instance][3] is public and had all mirror repos public as well. This
+caused a huge network traffic from bots. 
+
+I created an [organization][4] without public visibility and made it own all
+the mirror repos.
+
+```
+❯ yq --version
+yq (https://github.com/mikefarah/yq/) version v4.44.1
+
+❯ curl -V | head -c 11
+curl 8.8.0 
+```
+
+## API token
+
+## Hooks for notification
+[Hooks][5]
+
+Lets first download the list of all mirror repos
+
+```bash
+TOKEN=d88446542e844f4da4ba75bbb85bd694a71907b5
+
+curl "https://gitea.balki.me/api/v1/repos/search?limit=100&mode=mirror" \
+	-H "accept: application/json" \
+	-H "Authorization: token $TOKEN" \
+	-o mirror-repos.json
+```
+
+References
+
+1. API doc: https://gitea.balki.me/api/swagger#/repository/repoSearch
+
+Create a [hook][5] manually in one repo and get the hook using the API
+
+```bash
+curl -s "https://gitea.balki.me/api/v1/repos/MirrorWatch/snac2/hooks?page=1&limit=10" \
+        -H "Authorization: token $TOKEN" | yq -P -oj
+```
+
+Sample Output:
+```json
+[
+  {
+    "id": 32,
+    "type": "telegram",
+    "branch_filter": "tag",
+    "config": {
+      "content_type": "json",
+      "url": "https://api.telegram.org/bot1169894068:J1JVbV3f2vEQpdnPqFANfhjWZrFuUCJs1EW/sendMessage?chat_id=-1008910751069"
+    },
+    "events": [
+      "create"
+    ],
+    "authorization_header": "",
+    "active": true,
+    "updated_at": "2024-06-20T20:54:33-04:00",
+    "created_at": "2024-06-20T20:54:33-04:00"
+  }
+]
+```
+
+Now loop through all mirror repos and add the same webook. Remove unwanted fields like `id`, `created_at`, etc.,
+
+```bash
+yq -r '.data[] | .full_name' mirror-repos.json | while read -r repo; do
+	echo "$repo"
+
+	curl "https://gitea.balki.me/api/v1/repos/$repo/hooks" \
+		-H "Authorization: token $TOKEN" \
+		--json @- <<-EOM
+			{
+				"active": true,
+				"branch_filter": "tag",
+				"config": {
+					"content_type": "json",
+					"url": "https://api.telegram.org/bot1169894068:J1JVbV3f2vEQpdnPqFANfhjWZrFuUCJs1EW/sendMessage?chat_id=-1008910751069"
+                },
+				"events": [
+					"create"
+				],
+				"type": "telegram"
+			}
+		EOM
+
+	echo "============"
+done
+```
+
+## Fixing issues and pr links
+
+Fix the issue url setting in first repo as shown [here][7].
+
+Get the json representataion.
+
+```bash
+yq '.data[] | .external_tracker ' mirror-repos.json | head 
+```
+
+Sample output
+
+```json
+{
+  "external_tracker_url": "https://github.com/caddyserver/caddy/issues",
+  "external_tracker_format": "https://github.com/caddyserver/caddy/issues/{index}",
+  "external_tracker_style": "numeric",
+  "external_tracker_regexp_pattern": ""
+}
+```
+
+Now loop throug all repos and update. Making sure only add to github repos and
+they are not already updated
+
+```bash
+yq -r '.data[] 
+| select(.original_url == "*github*" and has("internal_tracker") )
+| "\(.full_name) \(.original_url)"' mirror-repos.json | while read -r repo og; do
+	echo "Repo is $repo and github origin url is $og"
+	curl "https://gitea.balki.me/api/v1/repos/$repo" \
+		-H "Authorization: token $TOKEN" \
+		-X PATCH \
+		--json @- <<-EOM
+			{
+					"external_tracker": {
+					"external_tracker_url": "${og%.git}/issues",
+					"external_tracker_format": "${og%.git}/issues/{index}",
+					"external_tracker_style": "numeric",
+					"external_tracker_regexp_pattern": ""
+					}
+			}
+		EOM
+done
+yq -r '.data[] | .full_name' mirror-repos.json | while read -r repo; do
+	echo "$repo"
+	jo has_actions=false | curl "https://gitea.balki.me/api/v1/repos/$repo" \
+		-H "Authorization: token $TOKEN" \
+		-X PATCH \
+		--json @-
+done
+```
+
+### Doc links
+* yq: [select][8], [has][9], [string interpolation][10]
+* bash: [parameter expansion][11], [here-doc][12]
+* curl: [`--json`][13]
+
+
+[1]: https://github.com/go-gitea/gitea
+[2]: https://docs.gitea.com/usage/repo-mirror#pulling-from-a-remote-repository
+[3]: https://gitea.balki.me
+[4]: https://docs.gitea.com/usage/permissions#organization-repository
+[5]: https://docs.gitea.com/usage/webhooks
+[6]: https://docs.gitea.com/development/api-usage
+[7]: https://github.com/go-gitea/gitea/issues/18986
+[8]: https://mikefarah.gitbook.io/yq/operators/select
+[9]: https://mikefarah.gitbook.io/yq/operators/has#select-checking-for-existence-of-deep-paths
+[10]: https://mikefarah.gitbook.io/yq/operators/string-operators#interpolation
+[11]: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
+[12]: https://www.gnu.org/software/bash/manual/html_node/Redirections.html#Here-Documents
+[13]: https://everything.curl.dev/http/post/json.html
+[14]: https://gitea.balki.me/api/swagger#/repository/repoSearch
+
--- a/layouts/_default/single.html
+++ b/layouts/_default/single.html
@ -21,4 +21,3 @@
        {{ .Content }}
    </article>
 {{ end }}
-
--- a/2
+++ b/2