ProcessMessages is a function in maXbox4 that is used to process messages sent to the maXbox4 application. It is responsible for handling the messages sent to the application, such as mouse clicks, key presses, and other user input. It also handles messages sent from the application to other parts of the system, such as the graphics engine or sound system. Switching off ProcessMessages! means more speed but less updated events in a message loop.
maXbox4 is a powerful and versatile tool for creating and managing digital content. It offers a wide range of features that make it easier to create, store, and share digital content. It also provides a secure platform for collaboration and communication. The main advantages of Maxbox4 include:
Easy to use interface: Maxbox4 has an intuitive user interface that makes it easy to create, store, and share digital content.
Secure platform: Maxbox4 provides a secure platform for collaboration and communication. It also offers a range of security features to protect your data.
Flexible storage: Maxbox4 offers a range of storage options, including cloud storage, local storage, and external storage.
Collaboration tools: Maxbox4 provides a range of collaboration tools, such as chat, file sharing, and task management.
Automation tools: Maxbox4 offers a range of automation tools, such as automated workflows and scheduled tasks.
Here we first define the function, then we call the main part in Python4maXbox:
eng.Execstr('import qrcode as pyqr');
eng.Execstr(QRCODE);
eng.Execstr('Qrcode_Maker("https://maxbox4.wordpress.com/")');
OpenDoc(exepath+'qrcodemx4.png');
Now you can create your own Qrcode with this awesome Python4maXbox script that will let you generate customize Qr Code by using the Qrcode module. This script is helpful when you need to share something or share text data in Qrcode form. Of course you has to install the module first:
1527 unit uPSI_dprocess; TProcess2 1528 unit uPSI_uXmlStorage.pas 1529 unit uPSI_AsphyreTimer.pas 1530 unit uPSI_Pas2JSUtils.pas 1531 unit uPSI_pacMain; (Form1pac: TForm1pac;) 1532 unit dwsWebUtils.pas; DWS 1533 unit uPSI_dwsWebUtils.pas; DWS beta 1534 unit uPSI_RestUtils2.pas; 1535 unit uPSI_Pas2jsFileUtils.pas; beta 1536 unit uPSI_JPerson.pas; 1537 unit uPSI_OldRttiMarshal; 1538 unit uPSI_superxmlparser; 1539 unit uPSI_superobject.pas; beta 1540 uPSI_NovusWindows.pas 1541 uPSI_NovusStringUtils.pas 1542 unit uPSI_NovusUtilities.pas; 1543 unit uPSI_NovusNumUtils; 1544 unit uPSI_NovusFileUtils; 1545 unit uPSI_NovuscURLUtils.pas; curl beta 1546 unit uPSI_uDM.pas; 1547 unit uPSI_dpipes.pas; 1548 unit uPSI_ShellAPI2; 1549 uPSI_NovusStringBuilder.pas 1550 unit NovusDateDiffUtil.pas 1551 unit NovusDateUtils; 1552 unit NovusDateStringUtils; 1553 unit uPSI_PJCBView; 1554 uPSI_NovusWinVersionUtils.pas 1555 unit uPSI_PJResFile.pas; 1556 unit PJResFile_Routines2; 1557 unit uPSI_JvCreateProcess2 1558 unit uPSI_JVCLHelpUtils.pas; 1559 unit uPSI_ModuleLoader; 1560 unit uPSI_JvLogClasses; 1561 uPSI_uExporter.pas; 1562 unit uExporterDestinationCSV.pas; 1563 uPSI_uOptionParser.pas 1564 uPSI_TOptionDefs; 1565 unit uPSI_GUIUtils.pas //dunit 1566 unit uPSI_GUIAutomation.pas; //dunit 1567 unit uPSI_GUIActionRecorder; //dunit 1568 unit TypeHelpers.pas //dunit 1569 unit uPSI_API_base; //API 1570 unit uPSI_API_audio; //API 1571 unit uPSI_API_ledgrid; 1572 uPSI_API_graphics.pas; 1573 uPSI_API_files.pas; 1574 uPSI_API_tools.pas; 1575 unit uPSI_API_winprocess; 1576 unit API_strings.pas; 1577 unit uPSI_API_services.pas
Image Classifier with loading and testing a pre-trained model by Max Kleiner
The Number guessing project
Debugging with the new debugger in Lazarus – lessons part 2
AI-enabled brain scanner reads thoughts
Lazarus compiling Delphi code
BLAISE PASCAL MAGAZINELIBRARY By Internet and on USB Stick The Library kit for BPM has been extended with new features: Search over ALL 111 issues and per issue.
Raize SoftWare DropMaster
Lazarus for Visual Studio
Delphi Community version for Delphi 11
Jim McKeeth leaving Embarcadero/Delphi
FastReport for Lazarus on LINUX in a Trial and as Professional version
Kalilinux 64-bit Osboxes.orgKalilinux 64-bit Osboxes.org with BBC Newsfeed API
The MindMaker AI Plugin is an open-source plugin that enables games and simulations within UE4 and UE5 to function as OpenAI Gym environments for training autonomous machine learning agents. The plugin facilitates a network connection between an Unreal Engine Project containing the learning environment, and a python ML library that receives data from Unreal Engine and parses into a custom OpenAI Gym environment for training the agent. The standalone machine learning library can either be a custom python script in the event you are creating your own ML tool using MindMaker’s Remote ML Server, or it could be a precompiled learning engine such as MindMaker’s DRL Engine(Stable-Baselines3 Algorithms). Regardless of which option you choose, using the MindMaker AI Plugin developers and researchers can easily train machine learning agents for 2D, 3D and VR projects.
#imports for open AI gym and a compatible machine library (stable baselines, neuroevolution etc)
import gym
import tensorflow as tf
from stable_baselines.deepq.policies import MlpPolicy as DqnMlpPolicy
from stable_baselines.deepq.policies import LnMlpPolicy as DqnLnMlpPolicy
from stable_baselines.deepq.policies import CnnPolicy as DqnCnnPolicy
from stable_baselines.deepq.policies import LnCnnPolicy as DqnLnCnnPolicy
from stable_baselines.sac.policies import MlpPolicy as SacMlpPolicy
from stable_baselines.sac.policies import LnMlpPolicy as SacLnMlpPolicy
from stable_baselines.sac.policies import CnnPolicy as SacCnnPolicy
from stable_baselines.sac.policies import LnCnnPolicy as SacLnCnnPolicy
from stable_baselines.td3.policies import MlpPolicy as Td3MlpPolicy
from stable_baselines.td3.policies import LnMlpPolicy as Td3LnMlpPolicy
from stable_baselines.td3.policies import CnnPolicy as Td3CnnPolicy
from stable_baselines.td3.policies import LnCnnPolicy as Td3LnCnnPolicy
BFH Diploma 2023 mindmaker visualsEKON27 ON
Bloom Filter
A Bloom filter is a space-efficient probabilisticdata structure, conceived by Burton Howard Bloom in 1970, that is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not – in other words, a query returns either “possibly in set” or “definitely not in set”. Elements can be added to the set, but not removed (though this can be addressed with the counting Bloom filter variant); the more items added, the larger the probability of false positives.
The high level idea is to map elements x∈X to values y=h(x)∈Y using a hash function h, and then test for membership of x’ in X by checking whether y’=h(x’)∈Y, and do that using multiple hash functions h.
{integer sylvester_next(const integer& n) {
return n * n - n + 1;
}
var
s,h : Tinteger;//UIntX.TIntX; // arbitrarily large integer
i : integer;
begin
s:= TInteger.create(1);
h:= TInteger.create(0);
//n*n-n+1
for i:= 0 to 9 do begin
//inc(s);
s.add1(1);
h.assign(s);
WriteLn( {SysUtils.}Format('s[%d] = %s',[i, s.ToString(normal)]));
//s := s*(s - 1);
s.mult(s);
s.subtract(h)
end;
s.free;
h.free;
WriteLn( 'Sum of reciprocals =');
//WriteLn( (s - 1).ToString);
WriteLn( '/'); // on a separate line for clarity
WriteLn( s.ToString(normal));
end.
Digitree puzzle
Place the digits 1 through 9 in the circles at right so that the value in any circle will be the sum of the digits in the circles connected from below.
the old exe and the new one scriptingthe way to solve
{************* LabelDragDrop ************}
procedure Label9DragDrop(Sender, Source: TObject; X, Y: Integer);
{dropping a label on a label..}
var i:integer;
begin
{ if the label is in its home column, then reset the label being dropped}
{if the label is in the figure, then reset the label being dropped on}
If TControl(sender).left=labelstart.x then resetlabel(TControl(source).tag)
else begin {the label we are dropping is already in a shape}
for i:=1 to 7 do {find the shape holdng the label we are dropping on}
if shapes[i].tag=TControl(sender).tag then begin
resetlabel(TControl(sender).tag); {move that label back home}
Shape1dragDrop(shapes[i],source,x,y); {and simulate a drop on that shape}
break; {might as well stop searching}
end;
end;
end;
Here is the second little puzzle program adapted from my most recent puzzle book acquisition, Giant Book of Mensa Mind Challenges , Stirling Publications, 2003.
The operation of the program is simple: Drag digits to the circles until the conditions are satisfied. Clearly, since there are nine digits and only seven circles, some of the circles will contain two digits. Drag and drop digits on the left or right side of the circle, depending whether you are setting the left or right digit.
TNeuralNetwork is a general purpose component to be used under Delphi to develop intelligent programs ranging from signal processing to intelligent internet applications. It uses a mixture of Artificial Neural Network Technology to converge to correct solutions with shorter training times. It is quite flexible and accepts different ranges of inputs and outputs defined by the user. Network allows any number of hidden layers and the structure of the network can be visually displayed.
Code reviews may be an important tool in the process of software development to ensure great code quality.Above all, they promote team exchanges on structure, style and shifts in the system.They also serve to reveal both improvements and weaknesses.With SonarQube we take a look at the five metrics Reliability, Security, Maintainability, Coverage and Complexity.And then we start with bug fixing and refactoring.
Code and tutorials for EKON 23 (rename archive *.key to *.zip)
Scripts like Shell Boy or prefer stewing in the Shell Hell?I show with Pascal Script and Delphi XE in maXbox how to extend each Delphi app native. With a script engine one (and also the customers themselves) one Adapt the application to your own requirements without recompiling. In addition, apps can be updated, tuned and tested by simply leaving a new script file as bytecode or in a batch production.
Python for Delphians Python is omnipresent, 30 years young and has now risen to the Big 5. Time to make a comparison and get to know the phenomenon of data science. Python code offers a few ways to communicate with C #, Delphi, or C ++ functions.Therefore, you can use Python as a script to integrate various functions developed in other languages into a compact program.We did that with the Pascal-Python pipeline using the Twitter Streaming API or TensorFlow.
Max Kleiner – Scripting like Shell Boy
Scripts like Shell Boy or prefer stewing in the Shell Hell?Max shows with Pascal Script and Delphi XE in the maXbox how each Delphi app can be extended native.With a script engine you (and the customers themselves) can adapt an application to their own requirements without having to recompile it. In addition, apps can be updated, tuned and tested by simply leaving a new script file as bytecode or batch production.
Skripten wie Shell Boy oder lieber schmoren in der Shell Hell? Max zeigt mit Pascal Script und Delphi XE in der maXbox wie sich jede Delphi App native erweitern lässt. Mit einer Skript-Engine kann man (und auch die Kunden selbst) eine Anwendung auf eigene Anforderungen anpassen, ohne neu zu kompilieren.
Darüber hinaus lassen sich Apps aktualisieren, tunen und testen, indem einfach eine neue Skriptdatei als Bytecode oder Batch die Produktion verlässt.
maxbox is also a small, standalone JS library that displays large images in a responsive, modal-style popup window. You’re allowed to customize the overlay background and button styles via HTML5 data attributes.
InmaXbox you can download, load and run directly scripts from the web. When setting the script execution policy ot the autoboot file, I generally first see what it is, then set the policy, and then check again to make sure that I have properly set the policy.
Your application update status maXbox has been updated on BytesIn.com to the 4.7.1 version. You can check out your application listed our site at: https://www.bytesin.com/software/maXbox/
If you want to let your users know about your software page on BytesIn to rate it, review it or share it, you can use our download banner below.
You have received this email because we have updated maXbox and this was the contact email supplied for the application.
When we talk about program execution, “asynchronous” means that the program doesn’t wait for a particular process to complete, but carries on regardless.
The big idea – Similar words tend to occur together and will have similar context for example – Apple is a fruit. Mango is a fruit. Apple and mango tend to have a similar context i.e fruit.
Sentiment Analysis refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. (Source: Wikipedia)
We’re going to train the neural network to do the following. Given a specific word in the middle of a sentence (the input word), look at the words nearby and pick one at random. The network is going to tell us the probability for every word in our vocabulary of being the “nearby word” that we chose.
In this session we propose 2 neural network models: CNN-LSTM and LSTM-CNN, which aim to combine CNN and LSTM networks to do sentiment analysis on Twitter data. We provide detailed explanations of both network architecture and perform comparisons against regular CNN, LSTM, and Feed-Forward networks. Finally, we find…
Max ist „gefühlte“ 60 Jahre am Coden und ist seit dem Sinclair ZX81 nie mehr davon losgekommen. Die Community kennt er seit der ersten EKON und nach diversen Studien und Arbeitgebern zu sicherheitsrelevanten Systemen wie auch als Lehrbeauftragter ist sein Pascal- und Python-Abenteuer ungebrochen. Auch PascalCoin, die maXbox, ModelMaker, OpenGL und OpenSSL sind ein Teil von ihm geworden und mit dem Internet of Things sowie dem Machine Learning gilt auch für ihn: Life is life long (l)earning.
60,445 new papers that discuss “Text Classification” have been added to Academia since you last searched for it.
Delphi in China Delphi has been operating in China since 1993 and currently employs approximately 28,000 people throughout Baicheng, Beijing, Changchun, Chengdu, Chongqing, Guangzhou, Nantong, Shanghai, Shenyang, Suzhou, Wuhan, Wuhu and Yantai. Delphi supplies the leading OEMs in China and is dedicated to providing innovative technology and superior service to its customers.
Conclusion to the Sentimiento Analizar Mood analysis and entity recognition are powerful social media analytics techniques for determining the context and semantic network of user content. La seguridad es uno de los temas, el libro and la emoción en la publicación. Envíenos mensajes a record and corpus tweets, a la vez a la API DE Twitter y a la API REST Twitter Streaming need. With the mood analysis we can classify texto en positivo, negativo y neutral categories. We have also learned that sentiment analysis has limited accuracy, especially for ambiguous expressions and incorrect grammar. For the sentiment analysis, we used the VDCER module (NLA VALUE Aware Dictionary for Sentiment Reasoning).
We have also seen that by machine learning on test and training data sets, we can create our own neural network sentiment analysis algorithm. The accuracy of the custom sentiment analysis depends on the quality, distribution and size of the example or training set. By building and applying our own mood analyzer with the Python Scikit Learn library and Keras / Tensorflow, we have an accuracy of around 71% de descuento. We have applied the crossvalidation, confusion matrix, K-fold, and precision / recall techniques to the performance of our algorithm and the CASSANDRA pipeline.
Fazit zur Sentiment Analyse Stimmungsanalyse und Entitätserkennung sind leistungsstarke Social-Media-Analysetechniken, um den Kontext und das semantische Netz von Benutzerinhalten zu ermitteln. Security ist ein Thema, das Gefühle und Emotionen beim Publikum hervorruft. Wir verwenden als Datensatz und Corpus Tweets, die die Twitter-API und die API REST Twitter Streaming benötigen. Mit der Stimmungsanalyse können wir Texte in positive, negative und neutrale Kategorien einteilen. Wir haben auch gelernt, dass die Sentimentanalyse mit Genauigkeit Einschränkungen aufweist, insbesondere bei mehrdeutigen Ausdrücken und fehlerhafter Grammatik.
It seems that we can map images to the percolation model by defining clusters on a binary representation for images. We show that critical percolating structures emerge in natural images and study their scaling properties by identifying fractal dimensions and exponents for the scale-invariant distributions of clusters. This formulation leads to a method for identifying clusters in images from underlying structures as a starting point for image segmentation.
Für die Stimmungsanalyse haben wir das VADER-Modul (Valence Aware Dictionary for Sentiment Reasoning) von NLTK verwendet. Wir haben auch gesehen, dass wir durch maschinelles Lernen an Test- und Trainingsdatensätzen unseren eigenen Stimmungsanalysealgorithmus mit einem neuronalen Netzwerk erstellen können. Die Genauigkeit der benutzerdefinierten Stimmungsanalyse hängt von der Qualität, Verteilung und Größe des Beispiels oder dem Trainingssatz ab.
Durch den Aufbau und die Anwendung eines eigenen Stimmungsanalysators mit der Python Scikit Learn-Bibliothek und Keras/Tensorflow haben wir eine Genauigkeit von rund 71% erhalten. Wir haben die Crossvalidation-, Confusionmatrix-, K-Fold- und Precision- / Recalltechniken auf die Leistung unseres Algorithmus und der CASSANDRA Pipeline angewendet.
Class output : Algorithms like SVM and KNN create a class output. For instance, in a binary classification problem, the outputs will be either 0 or 1. However, today we have algorithms which can convert these class outputs to probability. But these algorithms are not well accepted by the statistics community.
Probability output : Algorithms like Logistic Regression, Random Forest, Gradient Boosting, Adaboost etc. give probability outputs. Converting probability outputs to class output is just a matter of creating a threshold probability.
The chi-squared statistic measures the lack of independence between a feature (in this case, one term within a tweet) and class (whether the tweets are positive or negative). So far, in terms of feature extraction, I have tried four different methods: count vectorizer, Tfdif vectorizer, word2vec and doc2vec.
But the question remains: how do we make sure they’re meaningful? The answer: feed in pairs of words as a target word and a context word. Do this enough times, throwing in some bad examples too, and the neural network begins to learn what words appear together and how this forms almost a graph. Like a social network of words interconnected by contexts. ‘Good’ goes to ‘helpful’ which goes to ‘caring’ and so on. Our task is to feed this data into the neural network.
One of the most common approaches is the Skipgram model, generating these target-context pairings based on moving a window across a dataset of text. But what if our data isn’t sentences, but we still have contextual meaning? In this tutorial, our words are artist names and our contexts are genres and mean review scores. We want artist A to be close to artist B if they share a genre and have a mean review score that is similar. So let’s get started.
In the meantime I got another kernel to train music reviews.
What I want to show in the end is a correlation between genres and score with an autoencoder (target-context pairings based on genre and mean review score):
[[scores over genre—————genre – score – reviews count]]
genre - score - #reviews
electronic [6.8] 3874
metal [6.2,] 862
rock [6.7] 9438
None [6.5] 2371
rap [6.5,] 1559
experimental [8.1,] 1815
pop/r&b [7.1] 1432
folk/country [7.3] 685
jazz [7.1,] 435
global [7.0] 219
In the meantime we published the BBC news sentiment feed which is based on a RSS-Feed with 24 headlines per day in comparison for examples:
Title: 9: Covid: Three Democratic lawmakers test positive after Capitol riot : {“probability”: {“neg”: 0.4945 “neutral”: 0.7380 “pos”: 0.5054 , “label”: “neutral”}
Description: 9: ‘The Democrats say they sheltered in a safe room alongside others who refused to wear masks.’; {“probability”: {“neg”: 0.3661 “neutral”: 0.7792 “pos”: 0.6338 , “label”: “neutral”}
Link: 9: https://www.bbc.co.uk/news/world-us-canada-55631079 Three US lawmakers have tested positive for the coronavirus after sheltering for hours with colleagues during last week’s deadly assault on the Capitol. {“probability”: {“neg”: 0.5306 “neutral”: 0.7466 “pos”: 0.4693 , “label”: “neutral”}
Linktest without the word positive:
Three US lawmakers have tested for the coronavirus after sheltering for hours with colleagues during last week’s deadly assault on the Capitol. {“probability”: {“neg”: 0.5574 “neutral”: 0.7592 “pos”: 0.4425 , “label”: “neutral”}
Vader: neg neu pos compound human 9: 0.18 0.619 0.201 0.1779 slightly positive
import nltk
from nltk.sentiment.vader import SentimentIntensityAnalyzer
#import wget
import sys
nltk.download("vader_lexicon")
!pip install feedparser
import feedparser
import pandas as pd
pd.set_option('display.width',500)
pd.set_option('max_colwidth', 800)
import numpy as np
print("This first line after Imports")
def GraphViz(node):
d = Graph(node)
from graphviz import Digraph
dot = Digraph("Graph", strict=False)
dot.format = "png"
def rec(nodes, parent):
for d in nodes:
if not isinstance(d, dict):
dot.node(d, shape=d._graphvizshape)
dot.edge(d, parent)
else:
for k in d:
dot.node(k._name, shape=k._graphvizshape)
rec(d[k], k)
dot.edge(k._name, parent._name)
for k in d:
dot.node(k._name, shape=k._graphvizshape)
rec(d[k], k)
return dot
news_feed=feedparser.parse("http://feeds.bbci.co.uk/news/world/rss.xml")
sid = SentimentIntensityAnalyzer()
descriptions=["very negative","negative","slightly negative", \
"neutral","slightly positive", "positive","very positive"]
num_descriptions = len(descriptions)
frontpage = pd.DataFrame()
print(frontpage.info())
output_path = sys.argv[1]
def compound_polarity_descript(compound):
return descriptions[int(((1 + compound) / 2) * num_descriptions)]
#wget.download(url, out=destination) #, useragent= "maXbox")
for entry in news_feed.entries:
ss = sid.polarity_scores(entry.title + "\n" + entry.summary)
series = pd.Series(
[
ss["neg"],
ss["neu"],
ss["pos"],
ss["compound"],
compound_polarity_descript(ss["compound"]),
entry.title,
entry.summary
],
index=[
"neg",
"neu",
"pos",
"compound",
"human",
"title",
# print(eachItem["name"]," : ", eachItem["percentage_probability"])
"summary"
])
frontpage=pd.concat([frontpage,series.to_frame().T],ignore_index=True)
print(frontpage.info())
alist=[]
for count,entry in enumerate(news_feed.entries):
alist.append(str(count)+": "+entry.title)
finallist = "\n".join(alist)
with open(output_path, "w") as file:
file.write(str(finallist))
print(finallist)
compound_frontpage = frontpage["compound"].mean(skipna = True)
print("\n")
print(frontpage)
print("\n")
print("News today: "+compound_polarity_descript(compound_frontpage))
print("\n")
print("integrate newsfeed sentiment detector compute ends...")
And the snippet in Pascal maXbox from an REST-API:
function GetSentimentStream7(const S_API, pData: string): string;
var strm: TStringStream;
begin
//Function HttpPostURL(const URL,URLData: string; const Data:TStream):Boolean;
strm:= TStringStream.create('');
sr:='text='+HTTPEncode(pData)+CRLF;
sr:= sr+'language=english';
if HttpPostURL(S_API, sr, strm) then
result:= strm.dataString;
strm.free;
end;
IIndyHTTP:=TIdHTTP.create(self);
with TSimpleRSS.create(self) do begin
XMLType:= xtRDFrss;
IndyHTTP:= iindyhttp;
LoadFromHTTP(RSS_NewsFeed)
for it:= 0 to items.count-1 do begin
//Items[it].description
writeln(itoa(it)+': '+Items[it].title+' :'{+items[it].link});
println(ReplaceRegExpr('\d\d{4,12}[},]',
GetSentimentStream7(URLSentimentAPI2,Items[it].title),' ',false))
end;
println(items[9].link)
free;
IIndyHTTP.Free;
end; //}
end;
I did open another fork on guthub to document my adapted scripting units. Another advantage is the use of test-procedure with assertions. It implements the Assert procedure to document and enforce the assumptions you must make when writing code. Assert is not a real procedure. The compiler handles Assert specially and compiles the filename and line number of the assertion to help you locate the problem should the assertion fail.
Output is alike:
[nltk_data] Downloading package vader_lexicon to /root/nltk_data...
[nltk_data] Package vader_lexicon is already up-to-date!
Requirement already satisfied: feedparser in /usr/local/lib/python3.6/dist-packages (6.0.2)
Requirement already satisfied: sgmllib3k in /usr/local/lib/python3.6/dist-packages (from feedparser) (1.0.0)
This first line after Imports
<class 'pandas.core.frame.DataFrame'>
Index: 0 entries
Empty DataFrameNone
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 26 entries, 0 to 25
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 neg 26 non-null object
1 neu 26 non-null object
2 pos 26 non-null object
3 compound 26 non-null object
4 human 26 non-null object
5 title 26 non-null object
6 summary 26 non-null object
dtypes: object(7)
memory usage: 1.5+ KB
None
0: Trump impeachment: Republicans to join Democrats in House vote
1: Lisa Montgomery: US executes only woman on federal death row
2: Sinovac: Brazil results show Chinese vaccine 50.4% effective
3: 'Ndrangheta group: Italy braces for biggest mafia trial in decades
4: Philippines: Anti-'drug war' priest charged with sedition
5: YouTube suspends Donald Trump's channel
6: Huawei patent mentions use of Uighur-spotting tech
7: Shincheonji: Korean sect leader found not guilty of breaking virus law
8: YFN Lucci: US rapper wanted in Atlanta for suspected murder
9: Covid: Three Democratic lawmakers test positive after Capitol riot
10: Kelly Craft: US cancels top envoy's visit to Taiwan
11: Coon Cheese rebrands in Australia after anti-racism campaign
12: Indonesia coronavirus: The vaccination drive targeting younger people
13: CES 2021: Is this the world's smartest face mask?
14: Sheldon Adelson, the casino magnate who moved an embassy
15: Kuwait's stateless man who set himself alight
16: Viewpoint: Global media's Nigeria abductions coverage 'wrong'
17: Is seeing your doctor online working?
18: Capitol police officer Eugene Goodman hailed as 'a hero'
19: Uganda elections 2021: What you need to know
20: Silencing Trump: How 'big tech' is taking Trumpism offline
21: Covid-19: Alabama crowds ignore coronavirus to celebrate championship
22: What are differences the Covid vaccines?
23: Stammering: 'I thought it made me a failure'
24: Plastic surgery: Surgeons who target black women and use influencers for promotion
25: Why this female garbage truck driver hopes more women will join her
neg ... summary
0 0 ... A top House Republican says she will support impeachment in a vote expected to be held on Wednesday.
1 0.343 ... Lisa Montgomery, who killed a pregnant woman and took her baby, died by lethal injection in Indiana.
2 0 ... Latest results show Sinovac's Covid-19 vaccine is less effective in Brazil than previously suggested.
3 0.129 ... Hundreds of people will face a judge following a major inquiry into the 'Ndrangheta criminal group.
4 0.504 ... A vocal anti-"drug war" Catholic priest in the Philippines is to stand trial on charges of sedition.
5 0.286 ... The Google-owned service said the president had broken its rules over the incitement of violence.
6 0.109 ... The company denies selling technology that can identify the ethnic group and plans to reword the patent.
7 0.108 ... Lee Man-hee was however found guilty of embezzlement and handed a suspended jail term.
8 0.294 ... Police in Atlanta want to question YFN Lucci, 29, over a fatal shooting in the city last month.
9 0.18 ... The Democrats say they sheltered in a safe room alongside others who refused to wear masks.
10 0.228 ... The trip planned by Donald Trump's outgoing administration had infuriated China.
11 0.083 ... The 86-year-old name will disappear in Australia from July, after a campaign by Aboriginal groups.
12 0 ... Instead of inoculating the elderly, Indonesia is focusing on working-age adults. Will it succeed?
13 0 ... Gaming company Razer claims to have made the world's most technologically advanced mask
14 0 ... Sheldon Adelson, the son of Jewish immigrants who became one of the world's richest men, dies at 87.
15 0.224 ... Hamad had descended into despair, one of a Kuwaiti community denied rights for generations.
16 0.249 ... Novelist Adaobi Tricia Nwaubani criticises coverage of the kidnapping of schoolchildren.
17 0 ... The coronavirus pandemic has helped spark a boom in virtual doctor's visits.
18 0.169 ... Footage of the riot shows Mr Goodman lure a mob away from the Senate and the lawmakers hiding inside.
19 0 ... Five things you need to know about Uganda's divisive election.
20 0.112 ... Within days tech giants have moved to "deplatform" the president and some of his supporters.
21 0.159 ... Fans of the University of Alabama football team gathered in the streets of Tuscaloosa, ignoring social distancing.
22 0 ... Three vaccines have been approved in the UK - what are the differences between them?
23 0.298 ... Joe Biden will be the first known president with a stammer to take office.
24 0.118 ... A BBC investigation finds women suffered life-changing conditions after going to Turkey for surgery.
25 0 ... Belinda Janson hopes she won't be the only woman driving garbage trucks in Ipswich, Australia, for long.
[26 rows x 7 columns]
News today: neutral
integrate newsfeed sentiment detector compute ends...
Sentiment Peer Group Training
From script 1016_newsfeed_sentiment_integrate2.txt news training:
BBC-News Sentiment of 2021-07-05 09:17:25.544783 0: Miami collapse: Remaining structure demolished over safety fears: -0.3612 1: Afghanistan: All foreign troops must leave by deadline – Taliban: 0.34 2: Indonesia faces oxygen crisis amid worsening Covid surge: -0.8555 3: Watch: The moment dozens of fireworks accidentally explode: -0.5859 4: Dozens killed in Philippine military plane crash: -0.6705 5: Pope ‘responds well’ to colon surgery at Rome hospital: 0.2732 6: Israeli city’s residents fear for the future: -0.8689 7: Jacob Zuma: Supporters form ‘human shield’ to stop ex-president’s arrest: -0.3182 8: Cyprus: Nations send help to tackle worst wildfire in decades: -0.765 9: Indian dowry payments remarkably stable, study says: 0.3597 10: The questions raised by US government’s UFO report: 0.0 11: Didi says removal of app in China will affect business: -0.128 12: Is the use of jetpacks finally about to take off?: 0.2732 13: Letter from Africa: Lamenting the Algeria-Morocco border closure: -0.4588 14: Jeff Bezos steps down as Amazon boss: 0.1779 15: Nnamdi Kanu’s arrest leaves Nigeria’s Ipob separatists in disarray: -0.7184 16: ‘Coercion and rape’: Investigating my yoga school: -0.2023 17: Remaining collapsed Miami block demolished: -0.7269 18: Israeli videographer goes viral with sheep drone shots: 0.0 19: Portal: Lithuania and Poland build a window connecting two cities: 0.0 20: How dogs are helping the Surfside rescue effort: 0.7783 21: The cat with more than 50,000 social media followers: -0.128 2021-07-05 07:17:25.544783
function getOverallSentiment(feedstream: string; srec: TSentSumrecord;
details: boolean): TSentSumrecord;
var
Url,API_KEY, sr, flabel: string; fprob: double;
jo, locate: TJSONObject;
urlid: TIdURI;
strm: TStringStream;
begin
urlid:= TIdURI.create('');
Url:= urlid.URLEncode('http://text-processing.com/api/sentiment/');
strm:= TStringStream.create('');
sr:='text='+HTTPEncode(feedstream)+CRLF;
sr:= sr+'language=english';
if HttpPostURL(Url, sr, strm) then
jo:= TJSONObject.Create4(strm.dataString);
try
jo.getString('label')
locate:= jo.getJSONObject('probability');
//source:= jo.getJSONArray('results').getJSONObject(0).getString('source');
case jo.getString('label') of
'neg': begin fprob:= locate.getDouble('neg');
inc(srec.negct); end;
'neutral': begin fprob:= locate.getDouble('neutral');
inc(srec.neutct); end;
'pos': begin fprob:= locate.getDouble('pos');
inc(srec.posct); end;
end;
fLabel:= jo.getString('label');
srec.ftext:= feedstream;
srec.flabel:= flabel;
srec.fprob:= fprob;
if details then
srec.fdetails:=format('%2.3f %2.3f %2.3f',[locate.getDouble('neg'),
locate.getDouble('neutral'),
locate.getDouble('pos')]);
srec.fprobtot:= srec.fprobtot + fprob;
result:= srec;
except
//Xraise(Exception.Create(jo.getString('error')));
writeln('E: '+ExceptiontoString(exceptiontype, exceptionparam));
finally
jo.Free;
urlid.free;
strm.Free;
end;
end;
0: Hidden wealth of one of Putin’s 'inner circle' revealed
neutral prob: 0.670
1: Mariupol: Battle for key port city continues, Ukraine says
pos prob: 0.568
2: French elections: Macron targets Le Pen as run-off campaign begins
neutral prob: 0.814
3: 'IS Beatle' stood out from the others, court hears
neutral prob: 0.959
4: Marina Ovsyannikova: German outlet hires Russian protester
neutral prob: 0.901
5: Osinachi Nwachukwu: Nigeria gospel singer's husband arrested over death
neutral prob: 0.931
6: Pakistan: Shehbaz Sharif chosen as PM after week-long uncertainty
neutral prob: 0.892
7: Ukraine War: Russia warns Sweden and Finland against Nato membership
neutral prob: 0.925
8: 'I'm under no illusions,' says British soldier in Ukraine, after comrades killed
neutral prob: 0.823
9: Ayman Hadhoud: Questions over Egyptian economist's mysterious death
neutral prob: 0.545
10: NZ rugby players report body shaming and favouritism
neutral prob: 0.804
11: Southern Africa storms fuelled by climate change - study
neutral prob: 0.569
12: France election: This time it won’t be a walkover for Macron
neutral prob: 0.553
13: Ukraine village scarred by aftermath of occupation
neutral prob: 0.873
14: Somalia drought could kill 350,000 children - experts
neutral prob: 0.960
15: Saniya Mistri: The hijab-wearing teen rapper breaking stereotypes
neutral prob: 0.732
16: Chernihiv: Are these Russia's weapons of war?
neutral prob: 0.885
17: Ukraine: Three ways this war could escalate and drag Nato in
neutral prob: 0.837
18: Australia election: Enter a lacklustre advert for democracy
neutral prob: 0.798
19: Mozambique Palma terror attack: 'I can't go back'
neg prob: 0.730
20: López Obrador: What's behind Mexican leader's recall referendum?
neutral prob: 0.792
21: Inside Chernobyl: We stole Russian fuel to prevent catastrophe
neutral prob: 0.906
22: Ukraine war: The school that helped fleeing musicians find safety
neutral prob: 0.933
23: Zelensky asked if he'll give Russia any part of Ukraine
neutral prob: 0.795
24: Could Marine Le Pen win the French elections?
neutral prob: 0.733
25: Falklands War: 'The UK is still usurping our land'
neutral prob: 0.851
26: Ukraine war: The foreign fighters supporting the Ukrainian army
neutral prob: 0.791
27: Spanish police seize huge haul of illegal stuffed animals
neutral prob: 0.744
Overall: [1 / 26 / 1]
Overall Prob: [0.797]
mX4 executed: 11/04/2022 20:20:39 Runtime: 0:0:13.780 Memload: 39% use
And to compare with Vader
BBC-News Sentiment of 2022-04-11 20:33:32.575940 0: Hidden wealth of one of Putin’s ‘inner circle’ revealed: 0.4939 1: Mariupol: Battle for key port city continues, Ukraine says: -0.6597 2: French elections: Macron targets Le Pen as run-off campaign begins: 0.128 3: ‘IS Beatle’ stood out from the others, court hears: -0.7783 4: Marina Ovsyannikova: German outlet hires Russian protester: -0.25 5: Osinachi Nwachukwu: Nigeria gospel singer’s husband arrested over death: -0.7717 6: Pakistan: Shehbaz Sharif chosen as PM after week-long uncertainty: 0.0 7: Ukraine War: Russia warns Sweden and Finland against Nato membership: -0.6486 8: ‘I’m under no illusions,’ says British soldier in Ukraine, after comrades killed: 0.128 9: Ayman Hadhoud: Questions over Egyptian economist’s mysterious death: 0.3182 10: NZ rugby players report body shaming and favouritism: -0.2263 11: Southern Africa storms fuelled by climate change – study: -0.743 12: France election: This time it won’t be a walkover for Macron: 0.0 13: Ukraine village scarred by aftermath of occupation: -0.7351 14: Somalia drought could kill 350,000 children – experts: -0.9081 15: Saniya Mistri: The hijab-wearing teen rapper breaking stereotypes: 0.4215 16: Chernihiv: Are these Russia’s weapons of war?: -0.7003 17: Ukraine: Three ways this war could escalate and drag Nato in: -0.9095 18: Australia election: Enter a lacklustre advert for democracy: 0.0 19: Mozambique Palma terror attack: ‘I can’t go back’: -0.7579 20: López Obrador: What’s behind Mexican leader’s recall referendum?: 0.0 21: Inside Chernobyl: We stole Russian fuel to prevent catastrophe: -0.4404 22: Ukraine war: The school that helped fleeing musicians find safety: -0.5994 23: Zelensky asked if he’ll give Russia any part of Ukraine: 0.3378 24: Could Marine Le Pen win the French elections?: 0.5859 25: Falklands War: ‘The UK is still usurping our land’: -0.7506 26: Ukraine war: The foreign fighters supporting the Ukrainian army: -0.4939 27: Spanish police seize huge haul of illegal stuffed animals: -0.3182 2022-04-11 18:33:32.575940
To get ahead, just talk about things no one understands – even you!
erf: Whatcha doing? df: Working on a limited logistical contingency test. erf: Whats that? df: Well, basically it is a kind of systemized incremental stratification using synchronized monitored capability coupled with synchronized categorial concept. erf: Hows it going? df: Well, seem to be having a little trouble with ascendant logistical expectation. erf: I see. How long before you work the bugs out? df: Probably about a month. erf: Hey, that`s great. Keep me posted.
To get ahead, just talk about things no one understands - even you!
ssd: Whatcha doing? sdd: Working on a tabulated aggregational flexibility test. ssd: Whats that? sdd: Well, basically it is a kind of integrated cognition contingency using integrated convexity capability coupled with multiplex transitional rectification. ssd: Hows it going? sdd: Well, seem to be having a little trouble with systemized incremental catalyzation. ssd: I see. How long before you work the bugs out? sdd: Probably about a month. ssd: Hey, that`s great. Keep me posted.
To get ahead, just talk about things no one understands – even you!
sdsf: Whatcha doing? sd: Working on a regulated categorial time-phase test. sdsf: Whats that? sd: Well, basically it is a kind of participant platform variation using segmented incremental hardware coupled with synchronized management matrix. sdsf: Hows it going? sd: Well, seem to be having a little trouble with compatible mode construct. sdsf: I see. How long before you work the bugs out? sd: Probably about a month. sdsf: Hey, that`s great. Keep me posted.
To get ahead, just talk about things no one understands – even you!
zui: Whatcha doing? hjk: Working on a systemized logistical mobility test. zui: Whats that? hjk: Well, basically it is a kind of synchronized conversion projection using synchronized alterational mobility coupled with limited cognition options. zui: Hows it going? hjk: Well, seem to be having a little trouble with parallel management capability. zui: I see. How long before you work the bugs out? hjk: Probably about a month. zui: Hey, that`s great. Keep me posted.
Caption Catcher
* WORDS MAKE THE MAN *
To get ahead, just talk about things no one understands – even you!
kjh: Whatcha doing? hgj: Working on a regulated platform matrix test. kjh: Whats that? hgj: Well, basically it is a kind of systemized policy projection using parallel management capability coupled with integrated management stratification. kjh: Hows it going? hgj: Well, seem to be having a little trouble with ascendant transitional matrix. kjh: I see. How long before you work the bugs out? hgj: Probably about a month. kjh: Hey, that`s great. Keep me posted.
quoi qu`il en soit, encore une fois ?
Press any key to continue or Q to Quit
øøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøøø
Unsupervised learning via clustering can work quite well in a lot of cases, but it can also perform terribly. K-means is considered by many to be the gold standard when it comes to clustering due to its simplicity and performance.
K-means works by defining spherical clusters that are separable in a way so that the mean value converges towards the cluster centre. Because of this, K-Means may underperform sometimes. On the other side the Spectral clustering technique applies clustering to a projection of the normalized Laplacian. When it comes to image clustering, Spectral clustering works quite well.
As the algorithm tries to balance the volume (i.e. balance the region sizes), if we take circles with different sizes, the segmentation fails. Permutations are subsets selected from a set of objects in every possible order. That is, {1,2,3}, {1,3,2}, (2,1,3}, {2,3,1}, {3,1,2}, and {3,2,1} are all permutations of the set [1,2,3]. This listing the subsets is increasing alphabetical, also called Lexicographical Up.
In addition, as there is no useful information in the intensity of the image, or its gradient, we choose to perform the spectral clustering on a graph that is only weakly informed by the gradient. This is close to performing a Voronoi partition of the graph.
Another topic is PCA. Principal Component Analysis (PCA) is a simple yet popular and useful linear transformation technique that is used in numerous applications, such as stock market predictions, the analysis of gene expression data, and many more. By having natural language processing its more difficult than numerical data.
PCA yields the directions (principal components) that maximize the variance of the data, whereas LDA also aims to find the directions that maximize the separation (or discrimination) between different classes, which can be useful in pattern classification problem (PCA “ignores” class labels as Clustering also, means PCA and Clustering is unsupervised learning).
Next step is research of Text similarities. Finding cosine similarity is a basic technique in text mining. My purpose of doing this is to ope-rationalize “common ground” between PCA data reduction and similarities with the help of covariance matrix from PCA, some graphics to show that:
The similarity measure is the measure of how much alike two data objects are. Similarity measure in a data mining context is a distance with dimensions representing features of the objects. If this distance is small, it will be the high degree of similarity where large distance will be the low degree of similarity.
Skripten wie Shell Boy ————————————-
Skripten wie Shell Boy oder lieber schmoren in der Shell Hell? Ich zeige mit Pascal Script und Delphi XE in der maXbox wie sich jede Delphi App native erweitern lässt.
Mit einer Skript-Engine kann man (und auch die Kunden selbst) eine
Anwendung auf eigene Anforderungen anpassen, ohne neu zu kompilieren.
Darüber hinaus lassen sich Apps aktualisieren, tunen und testen, indem einfach eine neue Skriptdatei als Bytecode oder Batch die Produktion verlässt.
More concretely, PCA reduces data dimensions by finding principal components. These are directions of maximum variation in a dataframe. By reducing data original features or variables to a reduced set of new ones based on PCs, we end up with a min. number of variables that keep the max. amount of variation (information) about how data is distributed. If we end up with just two or three of these new variables, we will be able to represent each sample in our data in a 2D- or 3D-chart (scatterplot above and below).
After a half of year working with Python I want to spend some blue thoughts conclusions, start with NLTK of Natural Language Toolkit (like Spacy) to Sacred and TensorFlow with softmax() – see picture above, and ask How?::
LookupError:
************************************************
Resource stopwords not found.
Please use the NLTK Downloader to obtain the resource:
You see, the net is full of solutions and my mind is blowing with machine learning gone machine earning with latent or hidden features.
A highly practical example of latent variable models in machine learning is the topic modeling which is a statistical model for generating the words (observed variables) in the document based on the topic (latent variable) of the document. In the topic modeling, the words in the document are generated according to different statistical parameters when the topic of the document is changed. It is shown that method of moments (tensor decomposition techniques) consistently recover the parameters of a large class of latent variable models under some assumptions.
For a random vector, the first order moment is the mean vector, and the second order moment is the co-variance matrix (when the mean is zero). Higher order moments are usually represented using tensors which are the generalization of matrices to higher orders as multi-dimensional arrays.
Most of serial data (time series ) also hide the fact of auto-correlation. Auto correlation is a characteristic of data in which the correlation between the values of the same variables is based on related objects. It violates the assumption of instance independence, which underlies most of the conventional models. It generally exists in those types of data-sets in which the data, instead of being randomly selected, is from the same source.
On the other side Space-time patterns of rank concordance: Local indicators of mobility association with application to spatial income inequality dynamics for example. Concordance is a non-parametric method based on bootstrapping that is used to test the hypothesis that two subsets of time series are similar in terms of mean, variance or both. Simulations show that concordance is effective for detecting difference in both the level and scale of two sub-matrices containing non-stationary and dependent time series.
We work a lot with module collections:
This module implements specialized container data-types providing alternatives to Python’s general purpose built-in containers, dict, list, set, and tuple.
Here is an overview:
* namedtuple factory function for creating tuple subclasses with named fields
* deque list-like container with fast appends and pops on either end
* ChainMap dict-like class for creating a single view of multiple mappings
* Counter dict subclass for counting hashable objects
* OrderedDict dict subclass that remembers the order entries were added
* defaultdict dict subclass that calls a factory function to supply missing values
* UserDict wrapper around dictionary objects for easier dict subclassing
* UserList wrapper around list objects for easier list subclassing
* UserString wrapper around string objects for easier string subclassing
from nltk.corpus import indian
# NLTK reads the corpus as bytecodes.
hindi = ” “.join(indian.words(‘hindi.pos’))
bangla = ” “.join(indian.words(‘bangla.pos’))
marathi = ” “.join(indian.words(‘marathi.pos’))
telugu = ” “.join(indian.words(‘telugu.pos’))
# Prints out first 10 bytes (including spaces).
print ‘hindi:’, hindi[:10]
print ‘bangla:’, bangla[:10]
print ‘marathi:’, marathi[:10]
print ‘telugu:’, telugu[:10]
print
# Converts bytecodes to utf8.
hindi = hindi.decode(‘utf8’)
bangla = bangla.decode(‘utf8’)
marathi = marathi.decode(‘utf8’)
telugu = telugu.decode(‘utf8’)
# Prints out first 10 unicode char (including spaces).
print ‘hindi:’, hindi[:10]
print ‘bangla:’, bangla[:10]
print ‘marathi:’, marathi[:10]
print ‘telugu:’, telugu[:10]
print
And the list is getting longer to work with it:
C:\maXbox\mX46210\DataScience>pip3 list > pip3list2.txt
Algorithms
PyBrain implemented many useful standard and advanced algorithms in PyBrain, and in some cases created interfaces to existing libraries (e.g. LIBSVM). This list shows some of the methods you can use with PyBrain.
Plotting / Visualization
netCDF support
XML read/write
Variety of different dataset classes
Sacredboard is a web dashboard for the Sacred machine learning experiment management tool. It connects to the MongoDB database used by Sacred and displays a list of experiments, their state, Sacred configuration and the standard output from the running program.
and last from a top paper within 40 years:
Kotseruba & Tsotsos (2018). A review of 40 years cognitive architecture research.pdf
The emergent approach resolves the adaptability and learning issues by building massively parallel-models, analogous to neural networks, where information flow is represented by a propagation of signals from the input nodes.
However, the resulting system also loses its transparency, since knowledge is no longer a set of symbolic entities and instead is distributed throughout the network. For these reasons, logical inference in a traditional sense becomes problematic (although not impossible) in emergent architectures.
>>>>>>> Naturally, each paradigm has its strengths and weaknesses. For example, any symbolic architecture requires a lot of work to create an initial knowledge base, but once it is done the architecture is fully functional.
On the other hand, emergent architectures are easier to design, but they must be trained in order to produce useful behavior. Furthermore, their existing knowledge may deteriorate with the subsequent learning of new behaviors.
As neither paradigm is capable of addressing all major aspects of cognition, hybrid architectures attempt to combine elements of both symbolic and emergent approaches.
In psychology, the word subconscious (non symbolic, sub-symbolic, emergent) is the part of consciousness (symbolic) that is not currently in focal awareness. More at Wikipedia.
maXbox Starter 60 – Data Science with Machine Learning
“In the face of ambiguity, refuse the temptation to guess.”
– The Zen of Python
This tutor introduces the basic idea of machine learning with a very simple example. Machine learning teaches machines (and me too) to learn to carry out tasks and concepts by themselves. It is that simple, so above is an overview:
Of course, machine learning (often also referred to as Artificial Intelligence, Artificial Neural Network, Big Data, Data Mining or Predictive Analysis) is not that new field in itself as they want to believe us. For most of the cases you do experience 5-6 steps in different loops called Cognitive Chain:
• Collab (Set a control thesis, understand the data, get resources)
• Collect (Scrapy data, store, filter and explore data)
• Consolidate (normalization and aggregation, filters out irrelevant data)
• Cluster (Get a correlation model and category algorithm – unsupervised)
• Classify (Choosing an event model and classify algorithm – supervised)
• Conclude (Predict or report context thesis and drive data to decision)
For example, say your business needs to adopt a new technology in Sentiment Analysis and there’s a shortage of experienced candidates who are qualified to fill the relevant positions (also known as a skills gap).
You can also skip collecting data by your own and expose the topic straight to an Internet service API like REST to forward clustered data traffic directly to your server being accessed. How important collect, cluster and classify is points out next 5 definitions;
” Definition: Digital Forensic – to collect evidence.
” Taxonomy – to classify things.
” Deep Learning – to compute many hidden layers.
” Artificial Intelligence – brainless.
” XAI – Explainable Artificial Intelligence.
At its core, most algorithms should have a proof of classification and this is nothing more than keeping track of which feature gives evidence to which class. The way the features are designed determines the model that is used to learn. This can be a confusion matrix, a certain confidence interval, a T-Test statistic, p-value or something else used in hypothesis testing.
Lets start with some code snippets to grape the 5 steps, assuming that you have Python already installed (everything at least as recent as 2.7 should be fine or better 3.6 as we do), we need to install NumPy and SciPy for numerical operations, as well as matplotlib and sklearn for visualization:
“Collaboration”
import itertools
import numpy as np
import matplotlib.pyplot as plt
import maxbox as mx
from sklearn.decomposition import PCA
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from scipy import special, optimize
Then we go like this 5 steps:
• Collab (Python and maXbox as a tool)
• Collect (from scrapy.crawler import CrawlerProcess)
• Cluster (clustering with K-Means – unsupervised)
• Classify (classify with Support Vector Machines – supervised)
• Conclude (test with a Confusion Matrix)
“Collecting”
class BlogSpider(scrapy.Spider):
name = 'blogspider'
start_urls = ['https://blog.scrapinghub.com']
def parse(self, response):
for title in response.css('h2.entry-title'):
yield {'title': title.css('a ::text').extract_first()}
for next_page in response.css('div.prev-post > a'):
yield response.follow(next_page, self.parse)
print(next_page)
We are going to create a class called LinkParser that inherits some methods from HTMLParser which is why it is passed into the definition. This snippet can be used to run scrapy spiders independent of scrapyd or the scrapy command line tool and use it from a script also out of the shell.
“Clustering”
def createClusteredData(N, k):
pointsPerCluster = float(N)/k
X = []
y = []
for i in range (k):
incomeCentroid = np.random.uniform(20000.0, 200000.0)
ageCentroid = np.random.uniform(20.0, 70.0)
for j in range(int(pointsPerCluster)):
X.append([np.random.normal(incomeCentroid, 10000.0),
np.random.normal(ageCentroid, 2.0)])
y.append(i)
X = np.array(X)
y = np.array(y)
print('Cluster uniform, with normalization')
print(y)
return X, y
The 2 arrays you can see is X as the feature array and y as the predict array (array object as a list)! We create a fake income / age clustered data that we use for our K-Means clustering example above for the simplicity.
“Classification”
Now we will use linear SVC to partition our graph into clusters and split the data into a training set and a test set for further predictions.
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
# Run classifier, using a model that is too regularized (C too low) to see
# the impact on the results
It returns coordinate matrices from coordinate vectors. Make N-D coordinate arrays for vectored evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,…, xn.
Or just use predict for a given point:
print(svc.predict([[200000, 40]]))
print(svc.predict([[100000, 60]]))
print(svc.predict([[50000, 30]]))
print(svc.predict([[50000, 65]]))
>>> [3] [1] [1] [1]
“Conclusion”
The last step as an example of confusion matrix usage to evaluate the quality of the output on the data set. The diagonal elements represent the number of points for which the predicted label is equal to the true label, while off-diagonal elements are those that are mislabeled by the classifier. The higher the diagonal values of the confusion matrix the better, indicating many correct predictions.
def plot_confusion_matrix(cm, classes,
normalize=False,
title='Confusion matrix',
cmap=plt.cm.Blues):
"""
This function prints and plots the confusion matrix.
Normalization can be applied by setting `normalize=True`.
"""
if normalize:
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print("Normalized confusion matrix")
else:
print('Confusion matrix, without normalization')
print(cm)
A last point is dimensional reduction as the plot at first on the right corner down shows; its more a preparation but could also necessary to data reduction or to find a thesis.
Principal component analysis (PCA) is often the first thing to try out if you want to cut down number of features and do not know what feature extraction method to use.
PCA is limited as its a linear method, but chances are that it already goes far enough for your model to learn well enough.
Add to this the strong mathematical properties it offers and the speed at which it finds the transformed feature data space and is later able to transform between original and transformed features; we can almost guarantee that it also will become one of your frequently used machine learning tools.
This tutor will go straight to an overview to PCA.
The script 811_mXpcatest_dmath_datascience.pas (pcatest.pas) (located in the demo\console\curfit subdirectory) performs a principal component analysis on a set of 4 variables. Summarizing it, given the original feature space, PCA finds a linear projection of itself in a lower dimensional space that has the following two properties:
• The conserved variance is maximized.
• The final reconstruction error (when trying to go back from transformed
features to the original ones) is minimized.
As PCA simply transforms the input data, it can be applied both to classification and regression problems. In this section, we will use a classification task to discuss the method.
Although, PCA tries to use optimization for retained variance, multidimensional scaling (MDS) tries to retain the relative distances as much as possible when reducing the dimensions. This is useful when we have a high-dimensional dataset and want to get a visual impression.
Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it.
Machine learning algorithms can be divided into 3 broad categories — supervised learning, unsupervised learning, and reinforcement learning.Supervised learning is useful in cases where a property (label) is available for a certain dataset (training set), but is missing and needs to be predicted for other instances. Unsupervised learning is useful in cases where the challenge is to discover implicit relationships in a given unlabeled dataset (items are not pre-assigned). Reinforcement learning falls between these 2 extremes — there is some form of feedback available for each predictive step or action, but no precise label or error message.
To show that I produced a traceable example with 4 diagrams:
You see above a cluster and a classifier support vector machine
Next you see the confusion matrix with a PCA at the end:
Another Trace with more confusing elements <words / keywords> ratio:
And the last one out of the box:
Support Vector Machines: SVM is binary classification algorithm. Given a set of points of 2 types in N dimensional place, SVM generates a (N—1) dimensional hyperplane to separate those points into 2 groups. Say you have some points of 2 types in a paper which are linearly separable. SVM will find a straight line which separates those points into 2 types and situated as far as possible from all those points.
Clustering is used for analyzing data which does not include pre-labeled classes, or even a class attribute at all. Data instances are grouped together using the concept of “maximizing the intraclass similarity and minimizing the interclass similarity”. This translates to the clustering algorithm identifying and grouping instances which are very similar, as opposed to ungrouped instances which are much less-similar to one another. k-means clustering is perhaps the most well-known example of a clustering algorithm. As clustering does not require the pre-labeling of instance classes, it is a form of unsupervised learning, meaning that it learns by observation as opposed to learning by example.
Principal Component Analysis: PCA is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.
A good Book:
>>> Building Machine Learning Systems with Python
>>> Second Edition – Second edition: March 2015
maXbox source code is compiled into bytecode, the internal representation of a script in the mX4 interpreter. The bytecode is also cached in .psb files so that executing the same file is faster the second time (recompilation from source to bytecode can be avoided – menu…/Options/Save Bytecode¦Load Bytecode) or to run several times without recompilation on menu…/Program/Run Mode.
This “intermediate language” is said to run on a virtual machine that executes the machine code corresponding to each bytecode or take the same exe as a virtual machine on Wine (Wine 2.4 see pic below). Do note that bytecodes are not expected to work between different virtual machines (because different libraries / modules /policies), nor to be stable between releases.
Totals of Function Calls: 31475
SHA1: of 4.6.2.10 18200555A3407F8F7A40782618C9E4AE15590849
CRC32: 0113AD6D 27.1 MB (28,493,264 bytes)
Wine 5.0 stable was released 2 days ago with a large number of improvements. Here’s how to install it in Ubuntu 18.04, Ubuntu 19.10 (fix dependenices).
“Wine 5.0 is dedicated to the memory of Józef Kucia, who passed away in August 2019 at the young age of 30. Józef was a major contributor to Wine’s Direct3D implementation, and the lead developer of the vkd3d project. His skills and his kindness are sorely missed by all of us.”
I was wondering if there is an easy way to indent code in a IDE. Whenever I paste in code, all the code loses its indentation; what I have been doing is manually indenting every line with the space button. Is there a quicker way to do all of this?
Of course, such features would require choosing the source language of code block – this is another feature that is missing. You can learn by heart language: lang blocks, but it should be a menu option.
procedure TForm1Button1Click(Sender: TObject);var jsonToSend:TStringList;
http: TIDHttp;
begin
http:= TIDHttp.Create(nil);
try
http.HandleRedirects:= true;
http.ReadTimeout:= 5000;
jsonToSend:=TStringList.create;
jsonToSend.Add('{ Your JSON-encoded request goes here }');
Memo2.Lines.Text:=http.Post('http://my.restapi.url',jsonToSend);
finally
jsonToSend.Destroy;
http.free;
endend;
Of course also, there are esoteric languages where space is a part of syntax, not formatting (Python) so such feature must be optional and activated only by taskbar button.
When writing code in a code block I believe that it would be extremely useful if we could have the code auto indented instead of being forced to hit space bar multiple times. It would make editing so much faster. I tried this in maXbox but for the time beeing is just a simple indent guide to structure and find easier your code blocks:
Any reason this feature is not being considered for implementation? (about auto-indentation) This is a site for coders after all. Typing code into the editor is a common action to think what you want. Using an IDE to type in a few lines is not a good substitute. Comparing cost and benefit this feature seems to come out very well. This is what I call the Fleischmann Method (see below), to construct a locomotive and wait for passengers.
So before I speak I want to say something about 4.2.8.10.
From the perspective of the developer consuming a Web or Script API, everything at the other side of that interface is a sheer black box. Errors therefore become a key tool providing context and visibility into how to use an API.
The Release 8 has been born und the shine of Cologne with maXML and a lot of APIs
maXbox Connectors, powered by variant scripting, leverages maXbox Studio 4.2 technology to remove the complexity in connecting to multiple data sources in a single script file.
function ActiveDEP: Cardinal; uPSI_xrtl_util_ValueImpl.pas
procedure RichEditToCanvas(RichEdit: TRichEdit; Canvas: TCanvas; PixelsPerInch: Integer);
RichEditToCanvas(RichEdit1, Image1.Canvas, Self.PixelsPerInch);
Function CreateDOMNode( Doc : IDOMDocument; const NameOrData : DOMString; NodeType : TNodeType; const AddlData : DOMString) : IDOMNode’);
Function DetectCharEncoding( S : TStream) : TXmlEncodingType’);
Procedure CheckEncoding( var XMLData : DOMString; const ValidEncodings : array of string)’);
Function XMLStringToWideString( const XMLString : string) : WideString’);
Function FormatXMLData( const XMLData : DOMString) : DOMString’);
Function LoadXMLDocument3( const FileName : DOMString) : IXMLDocument’);
Function LoadXMLData( const XMLData : DOMString) : IXMLDocument;’);
Function LoadXMLData2( const XMLData : string) : IXMLDocument;’);
Function NewXMLDocument( Version : DOMString) : IXMLDocument’);
Procedure XMLDocError( const Msg : string);’);
Procedure XMLDocError2( const Msg : string; const Args : array of const);’);
function ProxyActive: Boolean;
function GetProxyServer: string;
function GetProxyOverride: string;
function GetProxyServerIP: string;
function GetProxyServerPort: Integer;
Totals of Function Calls: 31306
SHA1: of 4.2.8.10 C3FEF1DF49C22A312A249D5A59078CEE1E67E774
CRC32: 1D4C6F1F 26.9 MB (28,293,072 bytes)
procedure getHTTP_PNG(vimage: TImage; vURL: string);
var pngStream: TMemoryStream;begin
with TLinearBitmap.Create do try
pngStream:= TMemoryStream.Create;
try
HTTPget(vURL, pngStream)
except
showmessage(E.message)
end
pngStream.Position:= 0;
LoadFromStream2(pngstream,'PNG');
vimage.Picture:= NIL;
AssignTo(vimage.picture);
//SaveToFile(ExePath+'mX4_open.png');
finally
Dispose;
Free;
pngStream.Free;
end;end;
A WebSiteSniffer is a packet sniffer tool that captures all Web site files downloaded by your Web browser while browsing the Internet, and stores them on your hard drive under the base folder that you choose. WebSiteSniffer allows you to choose which type of Web site files will be captured: HTML Files, Text Files, XML Files, CSS Files, Video/Audio Files, Images, Scripts, and Flash (.swf) files.
The script below demonstrates how to get a computer’s IP address by the computer name using the ping command from WS and parsing its results with an RegEx.
Function ResolveIP(aPCName: string): string;
var objShell, objExec: oleVariant;
strOutput: string;
RegEx: TRegExpr; //TPerlRegex.create
begin
objShell:= CreateOleObject('WScript.Shell')
objExec:= objShell.Exec('ping '+aPCName +' -n 1 -4')
strOutput:= objExec.StdOut.ReadAll;
println(strOutput)
RegEx:= TRegExpr.create;
RegEx.Expression:= '(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})';
try
RegEx.Exec(strOutput)
// If RegEx.MatchPos[1] = 20 then
if RegEx.Match[1] <> '' then begin
writeln('regex match found');
result:= RegEx.Substitute('$&')
end else
result:= '""';
finally
objShell:= NULL;
objShell:= unassigned;
regEx.Free;
end;
End; // function
iset:= WMIExecQuery(isser,'Select * from Win32_OperatingSystem')
writeln(botoStr(WMIRowFindFirst(iset, ENum, tempObj)));
repeat
sysid:= ('OSid: '+vartoStr(tempobj.csname)+
' - '+vartoStr(tempobj.name)+
' -- '+vartoStr(tempobj.caption))
until not WMIRowFindNext(ENum, tempobj); //}
tempObj:= unassigned;
result:= sysID;
Output:
Sysid: OSid: MAXBOX8 -
Microsoft Windows 8.1|C:\WINDOWS|\Device\Harddisk0\Partition4
-- Microsoft Windows 8.1
As you may know a SHA (Secure Hash Algorithm) is one of a number of cryptographic hash functions. A cryptographic hash is like a signature for a text or a binary data file. SHA2 algorithm generates an almost-unique, fixed size 256-bit (32-byte) hash result.
Hash is a one way function – it cannot be decrypted or reversed back. This makes it suitable for password validation, challenge hash authentication, securisation, anti-tamper, digital signatures and of course blockchains.
In the following I want to show 2 solutions, one with the advapi32.dll and a second one with a library from PascalCoin (www.pascalcoin.org) precompiled in maXbox! Small functions to build a micro-service.
Such a hash you can find for example in anti-virus services to recognize a file already uploaded:
VirusTotal stores all the analyses it performs, this allows users to search for reports given an MD5, SHA1, SHA256 or URL. Search responses return the latest scan performed on the resource of interest. You see in the url above the SHA256 of the exe:
Already scanned files can be identified by their known (e.g. by default) SHA256 hash without uploading complete files. Now. lets build the script to get those hashes.
The DLL solution is not the easiest one but it shows explicitly steps behind. Also you do have the flexibility to use larger values like SHA512. Our goal is to calculate SHA256 of maXbox4.exe. First we need some types and structures:
The type TSHA_RES3 is kind of buffer for the 32-byte result in shaStr. I must admit that I managed to avoid pointers to pass so all the types are referenced and well managed. This seems very redundant but there is a very good default setting for all these parameter which makes sense for expressiveness, clarity and testing. On the other side PascalScript or Python cant handle pointers with one exception: PChar. And that was my helper. Next we define the const block:
Const
PROV_RSA_FULL = 1;
PROV_RSA_AES = 24;
CRYPT_VERIFYCONTEXT = $F0000000;
CRYPT_NEWKEYSET = $00000008;
// use with PROV_RSA_AES To get SHA-2 values.
//http://www.tek-tips.com/faqs.cfm?fid=7423
CALG_SHA256 = $0000800C;
CALG_SHA384 = $0000800D;
CALG_SHA512 = $0000800E;
HP_HASHVAL = $0002;
CRYPT32 = ‘crypt32.dll’;
MS_ENHANCED_PROV = ‘Microsoft Enhanced Cryptographic Provider v1.0’;
HASH256TEST= ‘The quick brown fox jumps over the lazy dog’;
By the way with OpenSSL and the well known libeay32.dll a further solution exists (just a type extract below) but in this article we focus on Win DLL.
type
SHA_CTX2 = Record
Unknown: Array[0.. 5] of LongWord;
State: Array[0.. 4] of LongWord;
Count: UInt64;
Buffer: Array[0..63] of Byte;
End;
function SHA256_CTX(nameform: DWord; namebuffer: array of char;
var nsize: DWord): boolean;
external 'SHA256@libeay32.dll stdcall';
function libeay32version: pchar;
external 'SSLeay_version@libeay32.dll stdcall';
Probably the best way to get started with this sort of thing is to create a small test DLL, create a few functions with known parameters and call it. In our case we need 6 functions to declare:
function CryptAcquireContext(out phProv: TCryptProv; szContainer:
PChar; szProvider: PChar; dwProvType: DWord;
dwFlags: DWord): boolean; //stdcall;
External 'CryptAcquireContextA@advapi32.dll stdcall';
function CryptCreateHash(phProv: TCryptProv; Algid: TAlgID; hKey:
TCryptKey; dwFlags: DWord; out phHash: TCrypthash): boolean;
External 'CryptCreateHash@advapi32.dll stdcall';
function CryptHashData(phHash: TCryptHash; aRes: PChar; dwDataLen:
DWord; dwFlags: DWord): boolean; //stdcall;
External 'CryptHashData@advapi32.dll stdcall';
function CryptGetHashParam(phHash: TCryptHash; dwParam: Dword;
out pbdata: TSHA_RES3;
var dwDataLen: DWord; dwFlags: DWord): Boolean; //stdcall;
External 'CryptGetHashParam@advapi32.dll stdcall';
function CryptDestroyHash(phHash: TCryptHash): Boolean; //stdcall;
External 'CryptDestroyHash@advapi32.dll stdcall';
function CryptReleaseContext(phProv: TCryptProv; dwFlags:DWord): boolean;
External 'CryptReleaseContext@advapi32.dll stdcall';
The quality of a DLL function is the parameter documentation. So much the better you find a well based documentation concerning view the parameter and return types of a function!
The Win module file format only provides a single text string to identify each function. There is no structured way to list the number of parameters, the parameter types, or the return type. However, some languages do something called function “decoration” or “mangling”, which is the process of encoding information into the text string.
Our first and important call is CryptAcquireContext():
The CryptAcquireContext function is used to acquire a handle to a particular key container within a particular cryptographic service provider (CSP). A CSP is an independent module that performs all cryptographic operations.
At least one CSP is required with each application that uses cryptographic functions. A single application can occasionally use more than one CSP. This returned handle is used in calls to CryptoAPI functions that use the selected CSP, so the first 2 calls are:
The following code assumes that the handle of a cryptographic context has been acquired and that a hash object has been created and its handle (hHash) is available. So we dont need any pointers and I can script it in maXbox, Python or Powershell with call by references and a strict PChar with the ByteArray
TSHA_RES3 = Array[1..32 ] of Byte;
The CryptCreateHash() function initiates the hashing of a stream of data.
This handle is used in subsequent calls to CryptHashData and CryptHashSessionKey to hash session keys and other streams of data that we get we a filetoString():
And the last step is to get the hash with CryptGetHashParam:
cbHashDataLen:= 32;
if (CryptGetHashParam(hHash, HP_HASHVAL, shares3,cbHashDataLen, 0))
then begin
for it:= 1 to cbHashDataLen do
shastr:= shastr +UpperCase(IntToHex((shares3[it]),2));
writeln('SHA256: '+shastr)
end;
I do always evaluate on each function the boolean return value to make sure. When was the last time you saw the return value for a function checked?
The CryptGetHashParam function retrieves data that governs the operations of a hash object. The actual hash value can be retrieved by using this function. Dont forget to free handles and structure:
The binToHEX_Str function is an effective way to get a HEX result test:
Const HexSymbols = '0123456789ABCDEF';
function binToHEX_Str(const bin: array of byte): string;
var i: integer;
begin
SetLength(Result, 2*Length(bin));
writeln('test length: '+itoa(length(bin)))
for i:= 0 to Length(bin)-1 do begin
Result[1 + 2*i + 0]:= HexSymbols[1+bin[i] shr 4];
Result[1 + 2*i + 1]:= HexSymbols[1+bin[i] and $0F];
end;
end;
Lets make an overview of the 6 functions used:
1. CryptAcquireContext Get handle to current key container of particular CSP.
2. CryptCreateHash Creates an empty hash object.
3. CryptHashData Hashes a block of data, adding it to spec. hash object.
4. CryptGetHashParam Retrieves a hash object parameter.
5. CryptDestroyHash Destroys a defined hash object.
6. CryptReleaseContext Releases handle acquired by the CryptAcquireContext().
By the way Indy retrieves SHA1 and with Indy 10:
function SHA1ADirect3(const fileName: string): string;
var fs: TFileStream;
begin
with TIdHashSHA1.Create do begin
fs:= TFileStream.Create(fileName,fmOpenRead);
try
result:= AsHex(HashValue(fs));
finally
fs.Free;
Free
end;
end;
end;
Next we step to the double SHA256 called SHA256D and block generation. Its important to realize that block generation is not a long, set problem (like doing a million hashes), but more like a lottery. Each hash basically gives you a random number between 0 and the maximum value of a 256-bit number (which is huge). If your hash is below the target, then you win. If not, you increment the nonce (completely changing hash) and try again to mine. With the SHA256 lib of PascalCoin the function is simpler to use in comparison to the DLL:
or more simpler with an alias in maXbox:
writeln(GetSHA256(sr))
function GetSHA256(Msg: AnsiString): string; //overload;
var Stream: TMemoryStream;
begin
Stream:= TMemoryStream.Create;
try
Stream.WriteBuffer(PAnsiChar(Msg)^,Length(Msg));
Stream.Position:= 0;
Result:= SHA256ToStr(CalcSHA256(Stream));
finally
Stream.Free;
end;
end;
Imagine now the double hash. It is also a crypto hash function, mainly used to ensure integrity of the encrypted message of the block, i.e. if you manipulate the message it will be visible, because the hash will also change. It also guarantees the uniqueness of a message or block of data.
In terms of Bitcoin or PascalCoin, it guarantees the uniqueness of each coin. So you cannot just copy the same set of data over and over again. The function is
Function CalcDoubleSHA256(Msg : AnsiString) : TSHA256HASH;
Function SHA256ToStr( Hash : TSHA256HASH) : String;
SHA256 (and thus SHA256D) is a cryptographic hash function (it performs a 1-way transformation on an input value) that forms the proof-of-work algorithm used when adding blocks to the blockchain in bitcoin. You are hashing the hexadecimal representation of the first hash. You need to hash the actual hash, the binary data that the hex represents.
Just semantics, but to avoid a common misunderstanding: SHA256 and others does hashing, not encoding. Encoding is something entirely different. For one it implies it can be decoded, whereas hashing is strictly a one-way (and destructive) operation!
There’s no guarantee that every single value in a hash function is reachable, depending on the hash algorithm. For some cryptographic algorithms, it is likely that less than half of the output keyspace is reachable for any given input. However, this may not hold true for every single cryptographic hash algorithm, and it is computationally unfeasible to verify.
There is also no proof that every output of common hash functions is reachable for some input, but it is expected to be true. No method better than brute force is known to check this, and brute force is entirely impractical.
maXbox 