Skip to content

remove venus module and migrate code to ceres, update README.md #507

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
genedna merged 1 commit into from
Aug 14, 2024
Merged

remove venus module and migrate code to ceres, update README.md #507

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ members = [
"common",
"mercury",
"jupiter",
"venus",
"ceres",
"libra",
"vault",
Expand All @@ -26,7 +25,6 @@ gateway = { path = "gateway" }
common = { path = "common" }
mercury = { path = "mercury" }
jupiter = { path = "jupiter" }
venus = { path = "venus" }
ceres = { path = "ceres" }
callisto = { path = "jupiter/callisto" }
gemini = { path = "gemini" }
Expand Down
29 changes: 17 additions & 12 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,18 +42,23 @@ For now, we are developing on the macOS and Arch Linux. And quick start manuel i

![Mega Architect](docs/images/Mega_Bin_and_Modules.png)

1. **mega** - The main entry of Mega.
2. **gateway** - The Gateway module is responsible for handling `git`, `git-lfs` and web UI requests through the HTTP and SSH protocol. More information on the [Gateway](gateway/README.md).
3. **libra** - The Libra is a `git` program that rewrite in Rust. More information on the [Libra](libra/README.md).
4. **craft** - The Craft is `git` filters include `git-lfs` and `encrypt`/`decrypt` filer. More information on the [Craft](craft/README.md).
5. **gemini** - The Gemini is a decentralized module of Mega. More information on the [Gemini](gemini/README.md).
6. **scorpio** - The Scorpio is a FUSE filesystem that allow you to mount a Mega repository as a local filesystem. More information on the [Scorpio](scorpio/README.md).
7. **mercury** - The Mercury module is the core module of Mega, which rewrites Git internal object like Blob, Tree, etc. More information on the [Mercury Module](mercury/README.md).
8. **ceres** - The Ceres implement translate protocol for Mega. More information on the [Ceres](ceres/README.md).
9. **jupiter** - The Jupiter is storage engine for Mega. More information on the [Jupiter](jupiter/README.md).
10. **venus** - The Venus is a monorepo engine for Mega. More information on the [Venus](venus/README.md).
11. **moon** - The Moon is a web UI for Mega. More information on the [Moon](moon/README.md).
12. **mars** - The Mars is website for Mega. More information on the [Mars](mars/README.md).
1. **mono** - Mega's api endpoint, responsible for handling `git`, `git-lfs` and web UI requests through the HTTP and SSH protocol.
2. **mega** - The main entry of Mega, parse cli command with clap.
3. **gateway** - The Gateway extend **mono** and contains ztm agent, which is responsible for ztm network capabilities. More information on the [Gateway](gateway/README.md).
4. **libra** - The Libra is a `git` program that rewrite in Rust. More information on the [Libra](libra/README.md).
5. **craft** - The Craft is `git` filters include `git-lfs` and `encrypt`/`decrypt` filer. More information on the [Craft](craft/README.md).
6. **gemini** - The Gemini is a decentralized module of Mega. More information on the [Gemini](gemini/README.md).
7. **scorpio** - The Scorpio is a FUSE filesystem that allow you to mount a Mega repository as a local filesystem. More information on the [Scorpio](scorpio/README.md).
8. **mercury** - The Mercury module is the core module of Mega, which rewrites Git internal object like Blob, Tree, etc. More information on the [Mercury Module](mercury/README.md).
9. **ceres** - The Ceres is Mega's Monorepo Engine and implement the git transport protocol. More information on the [Ceres](ceres/README.md).
10. **jupiter** - The Jupiter is storage engine for Mega. More information on the [Jupiter](jupiter/README.md).
11. **saturn** - The Saturn module implement user permissions with Cedar policy [Saturn](saturn/README.md).
12. **vault** - The Vault module is used for key and certificate management, implemented with RustyVault
13. **lunar** - The Lunar is a tauri app with Mega. More information on the [Lunar](lunar/README.md).
14. **aries** - The Mega's ztm Relay Server. More information on the [Aries](aries/README.md).
15. **altas** - AI API Module. More information on the [Altas](altas/README.md).
16. **moon** - The Moon is a web UI for Mono Engine. More information on the [Moon](moon/README.md).
17. **mars** - The Mars is website for Mega. More information on the [Mars](mars/README.md).



Expand Down
1 change: 0 additions & 1 deletion ceres/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ common = { workspace = true }
jupiter = { workspace = true }
callisto = { workspace = true }
mercury = { workspace = true }
venus = { workspace = true }
anyhow = { workspace = true }
tokio = { workspace = true, features = ["net"] }
tokio-stream = { workspace = true }
Expand Down
26 changes: 13 additions & 13 deletions ceres/src/api_service/import_api_service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ use mercury::errors::GitError;
use mercury::internal::object::commit::Commit;
use mercury::internal::object::tree::Tree;
use mercury::internal::object::tree::TreeItem;
use venus::import_repo::repo::Repo;

use crate::api_service::ApiHandler;
use crate::model::create_file::CreateFileInfo;
use crate::model::publish_path::PublishPathInfo;
use crate::protocol::repo::Repo;

#[derive(Clone)]
pub struct ImportApiService {
Expand All @@ -42,7 +42,7 @@ impl ApiHandler for ImportApiService {
async fn get_raw_blob_by_hash(&self, hash: &str) -> Result<Option<raw_blob::Model>, MegaError> {
self.context
.services
.mega_storage
.mono_storage
.get_raw_blob_by_hash(hash)
.await
}
Expand All @@ -58,9 +58,9 @@ impl ApiHandler for ImportApiService {
}
async fn get_root_commit(&self) -> Commit {
let storage = self.context.services.git_db_storage.clone();
let refs = storage.get_default_ref(&self.repo).await.unwrap().unwrap();
let refs = storage.get_default_ref(self.repo.repo_id).await.unwrap().unwrap();
storage
.get_commit_by_hash(&self.repo, &refs.ref_hash)
.get_commit_by_hash(self.repo.repo_id, &refs.ref_git_id)
.await
.unwrap()
.unwrap()
Expand All @@ -69,15 +69,15 @@ impl ApiHandler for ImportApiService {

async fn get_root_tree(&self) -> Tree {
let storage = self.context.services.git_db_storage.clone();
let refs = storage.get_default_ref(&self.repo).await.unwrap().unwrap();
let refs = storage.get_default_ref(self.repo.repo_id).await.unwrap().unwrap();

let root_commit = storage
.get_commit_by_hash(&self.repo, &refs.ref_hash)
.get_commit_by_hash(self.repo.repo_id, &refs.ref_git_id)
.await
.unwrap()
.unwrap();
storage
.get_tree_by_hash(&self.repo, &root_commit.tree)
.get_tree_by_hash(self.repo.repo_id, &root_commit.tree)
.await
.unwrap()
.unwrap()
Expand All @@ -88,7 +88,7 @@ impl ApiHandler for ImportApiService {
self.context
.services
.git_db_storage
.get_tree_by_hash(&self.repo, hash)
.get_tree_by_hash(self.repo.repo_id, hash)
.await
.unwrap()
.unwrap()
Expand All @@ -98,12 +98,12 @@ impl ApiHandler for ImportApiService {
async fn get_tree_relate_commit(&self, t_hash: &str) -> Commit {
let storage = self.context.services.git_db_storage.clone();
let tree_info = storage
.get_tree_by_hash(&self.repo, t_hash)
.get_tree_by_hash(self.repo.repo_id, t_hash)
.await
.unwrap()
.unwrap();
storage
.get_commit_by_hash(&self.repo, &tree_info.commit_id)
.get_commit_by_hash(self.repo.repo_id, &tree_info.commit_id)
.await
.unwrap()
.unwrap()
Expand All @@ -117,7 +117,7 @@ impl ApiHandler for ImportApiService {
) {
let storage = self.context.services.git_db_storage.clone();
let trees = storage
.get_trees_by_hashes(&self.repo, hashes)
.get_trees_by_hashes(self.repo.repo_id, hashes)
.await
.unwrap();
for tree in trees {
Expand All @@ -132,7 +132,7 @@ impl ApiHandler for ImportApiService {
) {
let storage = self.context.services.git_db_storage.clone();
let blobs = storage
.get_blobs_by_hashes(&self.repo, hashes)
.get_blobs_by_hashes(self.repo.repo_id, hashes)
.await
.unwrap();
for blob in blobs {
Expand All @@ -143,7 +143,7 @@ impl ApiHandler for ImportApiService {
async fn get_commits_by_hashes(&self, c_hashes: Vec<String>) -> Result<Vec<Commit>, GitError> {
let storage = self.context.services.git_db_storage.clone();
let commits = storage
.get_commits_by_hashes(&self.repo, &c_hashes)
.get_commits_by_hashes(self.repo.repo_id, &c_hashes)
.await
.unwrap();
Ok(commits.into_iter().map(|x| x.into()).collect())
Expand Down
8 changes: 5 additions & 3 deletions ceres/src/api_service/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,19 @@ use axum::async_trait;

use callisto::raw_blob;
use common::errors::MegaError;
use jupiter::utils::converter::generate_git_keep_with_timestamp;
use mercury::{
errors::GitError,
internal::object::{
commit::Commit,
tree::{Tree, TreeItem, TreeItemMode},
},
};
use venus::monorepo::converter;

use crate::model::{
create_file::CreateFileInfo, publish_path::PublishPathInfo, tree::{LatestCommitInfo, TreeBriefItem, TreeCommitItem, UserInfo}
create_file::CreateFileInfo,
publish_path::PublishPathInfo,
tree::{LatestCommitInfo, TreeBriefItem, TreeCommitItem, UserInfo},
};

pub mod import_api_service;
Expand Down Expand Up @@ -314,7 +316,7 @@ pub trait ApiHandler: Send + Sync {
}
}

let blob = converter::generate_git_keep_with_timestamp();
let blob = generate_git_keep_with_timestamp();
let mut last_tree = Tree::from_tree_items(vec![TreeItem {
mode: TreeItemMode::Blob,
id: blob.id,
Expand Down
42 changes: 22 additions & 20 deletions ceres/src/api_service/mono_api_service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,19 @@ use callisto::{mega_blob, mega_tree, raw_blob};
use common::errors::MegaError;
use jupiter::context::Context;
use jupiter::storage::batch_save_model;
use jupiter::utils::converter::generate_git_keep_with_timestamp;
use mercury::errors::GitError;
use mercury::hash::SHA1;
use mercury::internal::object::blob::Blob;
use mercury::internal::object::commit::Commit;
use mercury::internal::object::tree::{Tree, TreeItem, TreeItemMode};
use venus::import_repo::repo::Repo;
use venus::monorepo::converter;

use crate::api_service::ApiHandler;
use crate::model::create_file::CreateFileInfo;
use crate::model::mr::{MRDetail, MrInfoItem};
use crate::model::publish_path::PublishPathInfo;
use crate::protocol::mr::MergeRequest;
use crate::protocol::repo::Repo;

#[derive(Clone)]
pub struct MonoApiService {
Expand All @@ -39,7 +40,7 @@ impl ApiHandler for MonoApiService {
///
/// Returns `Ok(())` on success, or a `GitError` on failure.
async fn create_monorepo_file(&self, file_info: CreateFileInfo) -> Result<(), GitError> {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let path = PathBuf::from(file_info.path);
let mut save_trees = vec![];

Expand All @@ -55,7 +56,7 @@ impl ApiHandler for MonoApiService {
{
return Err(GitError::CustomError("Duplicate name".to_string()));
}
let blob = converter::generate_git_keep_with_timestamp();
let blob = generate_git_keep_with_timestamp();
let tree_item = TreeItem {
mode: TreeItemMode::Blob,
id: blob.id,
Expand Down Expand Up @@ -117,14 +118,14 @@ impl ApiHandler for MonoApiService {
async fn publish_path(&self, publish_info: PublishPathInfo) -> Result<(), GitError> {
let storage = self.context.services.git_db_storage.clone();
let repo: Repo = publish_info.into();
storage.save_git_repo(repo.clone()).await.unwrap();
storage.save_git_repo(repo.clone().into()).await.unwrap();
Ok(())
}

async fn get_raw_blob_by_hash(&self, hash: &str) -> Result<Option<raw_blob::Model>, MegaError> {
self.context
.services
.mega_storage
.mono_storage
.get_raw_blob_by_hash(hash)
.await
}
Expand All @@ -138,7 +139,7 @@ impl ApiHandler for MonoApiService {
}

async fn get_root_tree(&self) -> Tree {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let refs = storage.get_ref("/").await.unwrap().unwrap();

storage
Expand All @@ -152,7 +153,7 @@ impl ApiHandler for MonoApiService {
async fn get_tree_by_hash(&self, hash: &str) -> Tree {
self.context
.services
.mega_storage
.mono_storage
.get_tree_by_hash(hash)
.await
.unwrap()
Expand All @@ -161,7 +162,7 @@ impl ApiHandler for MonoApiService {
}

async fn get_tree_relate_commit(&self, t_hash: &str) -> Commit {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let tree_info = storage.get_tree_by_hash(t_hash).await.unwrap().unwrap();
storage
.get_commit_by_hash(&tree_info.commit_id)
Expand All @@ -176,7 +177,7 @@ impl ApiHandler for MonoApiService {
item_to_commit: &mut HashMap<String, String>,
hashes: Vec<String>,
) {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let trees = storage.get_trees_by_hashes(hashes).await.unwrap();
for tree in trees {
item_to_commit.insert(tree.tree_id, tree.commit_id);
Expand All @@ -188,15 +189,15 @@ impl ApiHandler for MonoApiService {
item_to_commit: &mut HashMap<String, String>,
hashes: Vec<String>,
) {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let blobs = storage.get_mega_blobs_by_hashes(hashes).await.unwrap();
for blob in blobs {
item_to_commit.insert(blob.blob_id, blob.commit_id);
}
}

async fn get_commits_by_hashes(&self, c_hashes: Vec<String>) -> Result<Vec<Commit>, GitError> {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let commits = storage.get_commits_by_hashes(&c_hashes).await.unwrap();
Ok(commits.into_iter().map(|x| x.into()).collect())
}
Expand All @@ -208,7 +209,7 @@ impl ApiHandler for MonoApiService {

impl MonoApiService {
pub async fn init_monorepo(&self) {
self.context.services.mega_storage.init_monorepo().await
self.context.services.mono_storage.init_monorepo().await
}

pub async fn mr_list(&self, status: &str) -> Result<Vec<MrInfoItem>, MegaError> {
Expand All @@ -220,13 +221,13 @@ impl MonoApiService {
vec![MergeStatus::Open, MergeStatus::Closed, MergeStatus::Merged]
// return Err(MegaError::with_message("Invalid status name"));
};
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let mr_list = storage.get_mr_by_status(status).await.unwrap();
Ok(mr_list.into_iter().map(|m| m.into()).collect())
}

pub async fn mr_detail(&self, mr_id: i64) -> Result<Option<MRDetail>, MegaError> {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let model = storage.get_mr(mr_id).await.unwrap();
if let Some(model) = model {
let mut detail: MRDetail = model.into();
Expand All @@ -238,7 +239,7 @@ impl MonoApiService {
}

pub async fn mr_tree_files(&self, mr_id: i64) -> Result<Vec<PathBuf>, MegaError> {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let model = storage.get_mr(mr_id).await.unwrap();
if let Some(model) = model {
let to_tree_id = storage
Expand Down Expand Up @@ -293,14 +294,15 @@ impl MonoApiService {
}

pub async fn merge_mr(&self, mr_id: i64) -> Result<(), MegaError> {
let storage = self.context.services.mega_storage.clone();
if let Some(mut mr) = storage.get_open_mr_by_id(mr_id).await.unwrap() {
let storage = self.context.services.mono_storage.clone();
if let Some(model) = storage.get_open_mr_by_id(mr_id).await.unwrap() {
let mut mr: MergeRequest = model.into();
let refs = storage.get_ref(&mr.path).await.unwrap().unwrap();

if mr.from_hash == refs.ref_commit_hash {
// update mr
mr.merge();
storage.update_mr(mr.clone()).await.unwrap();
storage.update_mr(mr.clone().into()).await.unwrap();

let commit: Commit = storage
.get_commit_by_hash(&mr.to_hash)
Expand Down Expand Up @@ -343,7 +345,7 @@ impl MonoApiService {
mut tree_vec: Vec<Tree>,
commit: Commit,
) -> Result<String, GitError> {
let storage = self.context.services.mega_storage.clone();
let storage = self.context.services.mono_storage.clone();
let mut save_trees = Vec::new();
let mut p_commit_id = String::new();

Expand Down
Loading