k depends on the failure potability of a single block, but by calculate it from n,l and p, we can ensure a stable quality l. In case of fail, you can inform the user and suggest the usage of an older file version.
I can not write the mathematic formulas here, but for every limit l > 0 near by 0 and n > 0 exist an k > 0 where the probability of an incomplete backup restore q is less than l. This is perfect, because all other classical backup program have also q > 0, which results from the probability of block data read error on backup media devices.
But the n private date CHK Data of one save set (Data of one backup session) can be added by ECC CHK. In my opinion, ECC shout by done bitwise. Amusing you have an ECC Algorithm where you can free determinate m = n + k, which means you have n Bits, store m Bits and you can lost k Bits during recovery without get some trouble. Red Solomon code family you can use in this way, for example
Why is ii possible? Some user files have ever a modification data much more older than the date of the last backups. If statistic says "0 incarnations", you know, here is a lost! From this basis you can calculate the lost probability p<1 of a single CHK. The User may need n CHK to store his private date part. p**n is, for moderate n, very near by 0! thats what you mean, if you say, is don't work
The most easy way to implement a special handling for users files is, to store them additional encrypted to some ftp space. For smaller amount of data, such kind of stuff is freely available on the Internet, for example in association to free web space. But this is alone not sabotage secure. By evaluation of statistic functions of user private files, the program can calculate the lost propability
I see, you acknowledge, there is enough space and that popular files are save in free net. Next step is, that the backup program mus be able to distinguish between popular and not popular, to handle the private file of an user in an other way. To do so, a backup programmer needs a statistic function, to determinate the popularity of a file. By the way, this may also interest a freesite master
Regular Backups, like Apples Time Machine for example, can reconstruct the situation every hour for one day, every day for one week, every week for one month and every month until medium is full. This means, backup software must be able to give a data set an out dating time, so the second hour backup on a day can store with a 24h lifetime. This gives data for a garbage collection on the nodes.
Even on persons who create publish stuff himself, i never see a "own content" part of more than 50%. And this are very big exeptions! The regular percentage of "own stuff" on the Hard disk capacity is round about 10% or less. So you must only open your mind to enhance free net to Handel his resources carefully while operate as backup.
But not every Hard disk is full, even if Murphy's law say, the HD's will be become full! On an TB HD, every body needs a lot of time to fill them. Hard disk's become more and more bigger, with means, the mean percentage of free space is increasing over time. Next point is the amount of "distributed stuff" in relation to "origin stuff". In my work of ISP, i often see customer computers.
You are right, if you say: i have a fixed number of users and i can construct a scenario, where the system runs over the disk space and the system becomes lossy. But this is not the realty of a popular application. Here you can and must use statistic term's to describe the situation. For example: We say, you need 2 Bytes of space for free net for every byte of regular space under protection.
Because I am an small ISP two, i have a good Idia what kind of software increases operation costs sand what software don't increase operating costs! The more of power consumption is less in Relation to the Power consumption of an additional Backup server! To understand, why it is possible to make it reliable, is quite more complex, because this needs scholastically theorems and data knowledge!
Costs: If it runs with background Priority at traffic and CPU power, then the system lives from an improvement of the percentage of resource using. Not every user have a Flatrate, but (in my country) nearly every user can get one. If free net prever's routing within an AS, then there are also no need for ISP's to protect their Backbones. In house traffic is the most profit traffic for an ISP!
No additional Power, means no additional CO2 pollution! Most of my customer did never use the capacity of there modern Hard disk in the lifetime of their computer, which means: p2p backup is a COST FREE powerfully solution to every computer user, if the cryptographic basic is ok, like in freenet. There must be only ensured, that it is traffic effective to protect ISP backbone links to avoid break
What did p2p-backup cost: (A) Traffic: on regular operation, it stores in the background. This means, it may use IPv6 background transport priority, created for applications like usenet. So this traffic did not interference with you regular INTERNET activity. On Flatrate, no Money need. You need 2x to 4x more storage on your HD, which cost 1 times a rarely small amount of money. No add. power!
The only question is, whats the needed relation beween backup to working storage. If you only have power user, who are creating new stuff, than you need 75% percent of a hardisk as backup and 25% as working area, to get space for multiple instances. But if you create a popular application, then all the users who are have 99% consuming, collected distribution stuff create a huge amount of space.
(5) is the hardest problem. You have the need to keep your backup locations confidential, which is not so easy if you have huge traffic. Or you rent an ultra hard storage, like backup storage in the Switzerland including an Army guarantee. That's rarely expansive! But you can also solve (1)-(5) by a p2p backup solution running on a basis like something like a not so big modified freenet software.
To protect against problem (4), you ave two options: You may (A) decide to put data in an special, fire resistant box or you may (B) copy the data to a location far away. (A) costs a lot of money and did not work automatically. If your business did not have the opportunity of different locations, you must rent space far away and upload every-kind, included all distributed stuff.
If you uses a second hard-disk, you can solve problem (3) by using a raid. If you are using a software who create and maintains a set of snapshots, you may be able to solve (1) and (2), in combination with a second hard-disk also (3). If you work with laptops, you have typical the problem, that the is only one internal disk. You may decide to use an additional sever to make automatic backups.
A good Backup system must protect against every kind of data loss. This are (1) User Mistakes, for example deleting some thing you still need (2) Software malfunction (3) Hardware malfunction (4) Fire, W hater, Storm ... and (5) Theft, Sabotage, Break-down, Robbery. A new Windows Computer comes for example with a hidden recovery partition. This solves particular problem (2) in relation to system
Only i use equipment for more $ for Backup then your project balance have money al the time i was visit free net website. By creating a no cost alternative (I have a flatrate internet port) free net have the opportunity to become a world file system. Huge traffic makes it also quite more heavy to analyse traffic data. As long freenet ist exotic, time conicidence Analysis is still a serius problem.
In my opinion, "keep in mind" should be a week type of insert. It don't create any new instances. Its less then insert or fetch, but more than no interest. This will only help hold intended to use content in storage, but is no substitution for real traffic on the content. Also free sites must be more complete, and knowledge about the fact that an object is still under ownership may help to do.
As result from the fact, most stuff will stored very often on freenet. Hundert Millions of Windows/Films/Games may stored in tousends of instances, cerating huge amount of space. By have an option to query object statistics (also helpfull for search), the backupsolution can determinate what stuff may be a "keep in mind" subject or is unique and must periodicly reinserted.
toad, you don't have any idea how powerfully backups will acquire space for free net. Assume the HD will split 50:50 between users space and free net as required for backup solution. No problem in times of TB HD's. The for every Byte there is a Byte on other backup users. But what people have on their computers? Its not Unique, most is distributed stuff like OS, Games, Open Sources, Films, ...
My RFE of the "Keep In Mind Messages" class was declined with the hint of SPAM. To split the "classical not used contet" who is designated to be overwritten in two groups of data where someone with is interested documented be KIMM and someone thats is truly not interested, is not rarely an improvement for Spammer - because they can only avoid the overwrite with respect to not used content.
This will limit the people who use freenet, and this limits the system Space and increases the system loss. So freenet becomes as content a set of pages containing more or less illigal content with a lot of missing data.
To develop some Major Legal Application, programs must get information about the probability of data loss. I have create one RFE here to do so.
I am thinking in the same direction. The evolution of Freenet is locked by an "Conspiration" Factor. Freenet is used in majority by people, who have some troble with laws. The rason for this is easy: it is quite more complex to use freenet instead of some web application.
Also Freenet is losy, wich makes it also not interesting. Freenet in the current configuration is only used, if needed.
2 votesAdmintoad (Admin, Freenet Project Inc.) responded
All Freenet nodes are “cache only”. Freenet is a cache! Opennet (connect to strangers) Freenet does have a small performance bias, so to some degree it will tend to connect to nearby nodes. However, in order for freenet routing to work well, we need mostly close connections and a few further away connections, in terms of routing location, which is not quite the same as network topology. One option for ISPs would be to offer a large node which users can connect to as a darknet peer. On darknet, it is hoped that many of a user’s friends will be on the same network or at least in the same country.
Is it acceptable to use optional a local scope well known multi cast message to find peers in the origin and neighborhood AS. The reach can limited by the TTL. At the moment, is quite risky because nearly every user have something to hide. But in case of more legal usage, there are user who are able to disclose the fact of using free net. But here we are back at the backup system discussion!
Well. Smaller ISP will get a hard problem with Free net, because for the individual users the probability is very low to get a significant amount of "in house" traffic. Every user will create the traffic of 20, or in the future more, links to other peers. Alternative solution: Run free net with ipv6 + background priority if the message was traveled over the first nodes. This prefers freecapacity.
Yes, this will be a major priority in the near future.
Encrypt your IP/DNS with the public key of the peer, Store it at /XSK@your_public_key/XSK@peers_public_key, give him secure your XSK@your_public_key. On accept, he uses /XSK@peers_public_key/XSK@your_public_key to ack with his enc IP/DNS. if both present open by using a cache to find some people. After validation with keys, you update your IP/DNS and query your other Peers. XSK b. no contet on key
Large freesites are not handled very efficiently at the moment. We have half-finished code to greatly improve this, but don’t have time to integrate it in the near future. Apart from that, it would be good to have a mirroring wizard.
With Freenet as distributed file system instead as communication system, this is trivial! simply cd /freenet/your_private_key@...../subdir and then wget from server or scp, rsync or cp from the NFS, SAMBA, .... mounted filesystem of the own web server. Please compare to the "Freenet Linux" Idia
233 votesAdmintoad (Admin, Freenet Project Inc.) responded
How would making a livecd or other form of distribution make Freenet a less complicated concept? It would be useful to have an easily deployable node however, using a USB stick or swap space… But this is out of scope for the project proper, maybe some volunteer will build it. And you will still need to complete the wizard in any case to specify your security settings, and you might need to specify which drives to use etc.
Within Linux, FUSE (http://fuse.sourceforge.net/) is the right basis. for example mkdir /freenet/newdir creates a new pair of user keys. One Dir is RO for the public site, one is a RW one for content management. A successful read to a public key of an other user will create an visible entry to the freenet directory like host names in SUN's /net auto-mounting folder. (autofs V4) Thats usefull!
In the past freenet was communicated as communication system, but it is more a distributed file system then an communication platform. In my opinion, the right support is file system to mount freenet! frost my be substitute by an local apache with /freenet as htdocs root point. Creating private key may be done with some specific mkdir. Insert management may be done internally; publishing by a cp
There has been significant progress on this feature, “transport plugins”, but Chetan’s work is not yet part of the main Freenet, and doesn’t yet support TCP; it only supports packet-like transports, so e.g. cloaking traffic as VoIP to avoid blocking. More work is needed, by me and Chetan, to get this working, but first steps have been taken.
If you use port 80 or 443, it must be valid http(s) traffic to be conform with transparent proxy caches. I have some customer who has trouble with a tax application of my gouverment, because this boy are running a vpn over a tcp with port 80 expecing this is alwais open in firewalls to reduce support amount. But a transpanert cache did not understand an crash the connection!
We will be able to get a rough idea of whether a file is retrievable when we implement random routed requests.