Gitlab升级14.10.0后运行出错 - 青浦海洋数码电脑城

[db:作者]

上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
) G: j6 ^$ ?; S; v4 Z( P' y7 ]登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
5 r6 ?$ O4 D' Q9 o! V) o3 }接着设置语言和本地化设置：
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
% a" V2 u0 T7 T0 L于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
$ `9 C' C  ]3 Q$ LRecipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
3 B( s% @: Y! d8 U  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run
      
& R, I) D6 C- o7 D5 T" a; o/ e      ================================================================================
5 F2 X5 e% k. p- D/ i1 O; I- ~! C      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
! I2 M9 u8 }1 G' D/ t0 X4 V      ================================================================================
  V2 t" E2 ^2 X. L% N, s      
8 @: A2 t8 V7 P/ p; q      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
, D$ r1 Y5 {6 o1 b      Command execution failed. STDOUT/STDERR suppressed for sensitive resource
! [9 y( Q" m  P; y3 Q      
      Resource Declaration:
      ---------------------
      suppressed sensitive resource output
      
      Compiled Resource:
. u/ y' H' t% @! Z/ v' X      ------------------
      suppressed sensitive resource output
* n: z7 Z& R4 f3 I# z7 f      
, H/ S. ^; E: @% ?$ |      System Info:
- ^' @, H# N+ `3 F# t0 O' X) ^$ P      ------------
4 E+ c: d8 x3 s      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
+ f/ s' W3 E' m, v      program_name=/opt/gitlab/embedded/bin/chef-client
; ^5 Z& }6 H" O      executable=/opt/gitlab/embedded/bin/chef-client
      
   
    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================
   
    Mixlib::ShellOut::ShellCommandFailed
- H/ C  e, M! e2 e7 N) _0 E8 Z    ------------------------------------
3 c! z! Y- \4 j+ L8 J    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
! b, Y  n" v7 F4 W   
    Resource Declaration:
  i( a1 \2 F. [6 P5 [    ---------------------
$ ?/ b) ?5 |/ V- h    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
   
     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
2 @) Y" O- @# Y4 e/ p     54:   helper migration_helper
     55:
+ m3 N  N+ a0 L, ]     56:   environment env_variables
0 B9 O" y! _8 E  b     57:   dependent_services dependent_services
) z) t* {8 [; ~     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
5 E, u" |- P( x" A) L' W     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
" Z- `" x) _2 c/ D: b3 ^, Z     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
   
    Compiled Resource:
" `5 f/ a" ?8 ^( D- R; |, I    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
' D* S# u" U7 D/ P9 A& \   
- P1 z" n6 l* }/ T9 U  @9 e    rails_migration("gitlab-rails") do
" C$ h4 Q* \, m6 |; U0 @  H      action [:run]
! I  z5 U) l1 {8 ~      default_guard_interpreter :default
+ V% M+ C) }9 `% @- j      declared_type :rails_migration
0 V1 j5 n- r% X1 X3 P8 h* M      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
" c. s7 B5 i/ Q" y! W      logfile_prefix "gitlab-rails-db-migrate"
4 k6 d7 s4 a1 e& l8 X4 x2 ^+ q, }      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
' r7 k' o. c2 \      dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
      only_if { #code block }
    end
   
  f0 b+ K! i- P9 t9 u  O0 W    System Info:
4 @9 g! v% Q- f: h* [; F    ------------
    chef_version=15.17.4
- ]% b+ A# O5 p' i+ N    platform=ubuntu
    platform_version=20.04
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
1 w7 \1 o* D" c. D$ @$ |* t" }* x   
Running handlers:
There was an error running gitlab-ctl reconfigure:
  n, e0 w5 {6 _: [$ [9 orails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Running handlers complete
# t' V, d: h/ P: e8 [2 p0 dChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；
5 C, n- ]9 T! E4 P* g% G2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
, m: c1 C9 F6 e" M8 z6 D/ |3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
5 ?6 j+ D  E3 r& ~0 s: p4. 运行 gitlab-ctl reconfigure;
1 ]- K$ m$ J. e* [5. 重启Gitlab：gitlab-ctl restart。
, }; M- a4 r% E7 I需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
2. gitlab-ctl reconfigure failing due to migration issue
AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2)