flink-cdc 同步到StarRocks 报错整理

flink-cdc 同步多张mysql分表到StarRocks 集群的一张表里,最终同步成功,中途遇到几次报错,在这里记录一下,以供参考:

1、 报错:
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalArgumentException: Couldn’t get the sink table’s column info.

【解决办法】:smt 生成的建表语句建完数据库之后要授权给正在使用的非root 的mysql 账号

2、 执行flink-sql 报错
Unable to deploy mysql debezium connector on GKE - User does not have the ‘LOCK TABLES’ privileges

【解决办法】:需要在建表语句添加:
‘debezium.snapshot.locking.mode’ = ‘none’

3、 执行flink-sql 报错
还是报错:Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation

【解决办法】:需要找运维修改账号权限:
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON . TO ‘user’ IDENTIFIED BY ‘password’;
FLUSH PRIVILEGES;

4、 刚开始报错找不到be节点
1064 - fail to create tablet: timed out. unfinished replicas(1/1): 49073(X.X.X.X) timeout=10s
时间: 10.027s

【解决办法】:重启be解决

最后终于启动起来,实现实时同步!!!

【版本信息】:
flink-1.13.3
flink-sql-connector-mysql-cdc-1.4.0.jar
flink-connector-starrocks-1.1.13_flink-1.13.jar

2赞

关于第二点,建议使用Flink CDC 2.0 之后的版本,锁表是个可能影响数据库生产环境的操作,
而这个 ‘debezium.snapshot.locking.mode’ = ‘none’ 参数会将Flink CDC 的读取语义从exactly-once 退化为at-least-once, Flink CDC 2.0 之后可以不锁表且保持exactly-once语义

2赞

好的,我后面升级一下试试,多谢~